Mod player in Rust - part 2

My last article finished with reading the pattern tables from the mode files. In this post I want to finish parsing the entire file so we can move onto playing it. First we need to work out where the pattern data is and how long it is.

Checking the format

The pattern data is located after the pattern table and before the sample data ( so far we have only read the sample information but not the actual sound data ) but there are a couple of complications;

1. The pattern table might be followed by a format tag which describes how many samples and channels the file contains. If it is a known tag we should just skip it. The oldest versions of the MOD file do not have a format tag. If we dont recognize a tag it is either because the file does not have one and is in the original format or it is a new one that we dont recognize.
2. The number of patterns is not described anywhere but the size of the pattern data is worked out as by deducting the size of everything else from the files size. What remains must be pattern data. Because each pattern has 64 notes for each channel and each note (I will get to them) is 4 bytes, the size of one patterns is;

pattern_size = 64 * 4 * num_channels

We can use the the pattern_size to check our assumptions about the mod file; Taking the modulo pattern_size from pattern_data_size should always be zero.

Given the complexities around the format tag I have decided to change the function identify_num_samples to get_format which will attempt to identify the tag and derive the number of channels and samples from it. The new structure and function are below;

struct FormatDescription{
    num_channels : u32,
    num_samples : u32,
    has_tag : bool      // Is the format description based on a tag
}

fn get_format(file_data: &Vec<u8> ) -> FormatDescription {
    let format_tag = String::from_utf8_lossy(&file_data[1080..1084]);
    match format_tag.as_ref() {
        "M.K." | "FLT4" | "M!K!" | "4CHN" => FormatDescription{ num_channels : 4, num_samples : 31, has_tag : true },
        _ => FormatDescription{ num_channels : 4, num_samples : 15, has_tag : false }
    }
}

The only new bit of syntax in the above code is the use of | to list all the matches that will get the same result.

With the new format description we can rewrite the sample reading loop to use it and skip the tag if we think it is present.

    let format = get_format(&file_data);
    for _sample_num in 0..format.num_samples {
    //... read the sample info
    //..
    // Skip the tag if one has been identified
    if format.has_tag {
        let format_tag = String::from_utf8_lossy(&file_data[offset..(offset + 4)]);
        offset += 4;
    }

Before working out the space left for the patterns we need to check how much space is used by the sample data.

    let mut total_sample_size = 0;
    for sample in &mut samples {
        total_sample_size = total_sample_size + sample.size;
    }

Now we finally have the information required to work out the pattern size and to check our assumptions about the mod file we are reading.

    let total_pattern_size = file_data.len() as u32  - (offset as u32) - total_sample_size;
    let single_pattern_size = format.num_channels *  4 * 64;
    let num_patterns = total_pattern_size / single_pattern_size;
    if total_pattern_size % single_pattern_size != 0 {
        panic!( "Unrecognized file format. Pattern space does not match expected size")
    }

The total_pattern_size is all the that remains after we deduct what has already been read and what we know will be taken up by the samples.

The single_pattern_size is the size of one pattern which depends on the number of channels.

Dividing the two will give us the number of patterns in the file. We finally do a sanity check by taking the modulo and checking that is is zero ( which tells us that there are no bytes that are unaccounted for )

Notes

The basic pattern building block is a note which specifies which sample to play, its period and any effects. The data is packed into 4 bytes in the following manner

byte0       byte1       byte2       byte3
76543210    76543210    76543210    76543210
SSSSPPPP    pppppppp    ssssEEEE    AAAAAAAA

SSSSssss      = sample number
PPPPpppppppp  = period
EEEE          = effect
AAAAAAAA      = effect argument

We get the sample number by combining the high bits from byte 0 and low bits from byte 2.The very first mod could only have 15 samples so only the low bits in byte 2 mattered.

The period is a reference to how many clock ticks between sending each byte of data to the digital-to-analog converter. Because the format originated on the Amiga this is with reference to its clock tick frequency. We will convert this into sample rate but for now we will just store the period. The low bits of byte 0 are bite 11-8 and the rest is stored in byte1.

The effect and its argument can be be extracted with similar bit manipulation. Below the Note structure and its constructor. The only new bit of Rust coding here is the structure initializer. When the variable name and structure field names match we can use abbreviate initializer syntax Struct{ field, field12...} instead of needing to write Strcut{ field = field, field2 = field2 }.

struct Note{
    sample_number: u8,
    period: u32,
    effect: u8,
    effect_argument: i8,
}

impl Note{
    fn new( note_data : &[u8]) -> Note {
        let sample_number = (note_data[2] & 0xf0) >> 4;
        let period = ((note_data[0] & 0x0f) as u32) * 256 + (note_data[1] as u32);
        let effect = note_data[ 2] & 0x0f;
        let effect_argument = note_data[3] as i8;
        Note{
            sample_number, period, effect, effect_argument
        }
    }
}

Converting numbers to enums

The above works but it would be better to store the effect as an enum rather than a number. So I created the following enum for all the effects.

enum Effect {
    Arpeggio = 0,
    SlideUp = 1,
    SlideDown = 2,
    //....all other effects
}

I thought I could simply cast the u8 into an Effect but I would get the compile error non-primitive cast: `u8` as `Effect`

Given Rust's focus on safety this restriction seems sensible enough as a simple type cast could potentially cause the number to cast into an undefined value. But it is also very annoying and I was hoping to find some built-in functionality that would check the value against different variants on the enum and return Err on failure but I have not found any.

Thinking about it a bit more I realized I am not really making full use of the Rust enums. Instead of converting the effect number into an enum value I should combine it with the effect argument. This is exactly the sort of thing the Rust enums are designed for. So I changed my effect enum to;

enum Effect{
    ..//
    VolumeSlide{ volume_change_per_tick : i8 }, // 12
    PositionJump{ pattern_table_pos : u8 },     // 13
    SetVolume{ volume : u8 },           // 14
    SetSpeed{ speed : u8 },             // 15
}

and I set up an Impl block for the Effect to handle the conversion from effect_number and effect_argument into an Effect

impl Effect{
    fn new( effect_number : u8, effect_argument : i8 ) -> Effect {
        match effect_number  {
            0 => match effect_argument {
                0 => Effect::None,
                _ => panic!( format!( "unhandled arpeggio effect: {}", effect_number ) )
            },
            // ...                
            14 => Effect::SetVolume{ volume : effect_argument as u8 },
            15 => Effect::SetSpeed{ speed : effect_argument as u8 }, 
            _ => panic!( format!( "unhandled effect number: {}", effect_number ) )
        }
    }
}

This feels better as now the Effect captures the effect type and the data associated with it. Because the conversion happens through the match syntax I have to deal with effects that I can't handle yet. For now I am panicking because I want to quickly identify mods that the code cant handle but I can imagine changing it to either ignore or report the unhandled effect.

My only complaint about this solution is that the link between the Effect and its numerical representation is only captured in the new function. It would be nicer if it could somehow be part of the enum definition.

The interesting new bit of code is the use of a nested match statement. In the mod file the arpeggio effect is only an effect if it has a non-zero argument. If it is zero it is basically a no-op.

Reading the patterns

Patterns are 64 lines long and store the note data for every channel. This gives us the following structure for storing the pattern data and the Pattern constructor for constructing an empty pattern without any note data;

pub struct Pattern {
    channels: Vec<Vec<Note>>       // outer vector is the lines (64). Inner vector holds the notes for the line             
}

impl Pattern{
    fn new( ) -> Pattern {
        let mut channels : Vec<Vec<Note>> = Vec::new();
        for _channel in 0..64 {
            channels.push( Vec::new() );
        }
        Pattern{ channels }
    }
}

The data is parsed a note and line at a time and used to set the pattern and finally we read the sample data.

    for sample_number in 0..samples.len() {
        let length = samples[sample_number].size;
        for _idx in 0..length {
            samples[sample_number].samples.push(file_data[offset] as i8);
            offset += 1;
        }
    }

It is a bit ugly because we convert each u8 into a i8 in a loop. I imagined there would be standard slice conversion functions but I can't find any. I am sure there are unsafe ways of doing it but that sort of defeats the point of using Rust.

In my next article I will look into audio and threads with Rust in preparation for playing out the mod music.I have uploaded all the code to https://github.com/janiorca/articles/tree/master/mod_player-2

Mod player in Rust - part 1

For my next Rust project I want to try something a bit more challenging than the Sudoku solver. I want to write a mod player in Rust

I learned my programming skills by writing demos for the Amiga 500 in 68000 assembler in the early nineties. I knew all about cycle counts, register optimizations and cost of memory access before I knew what a structure was.

Practically All the demos had music which were provided by mod players. At the time I didn't pay much attention to how the mod players were written but given Rust's focus on low level programming I think this would be a good little project for getting deeper into Rust.

Mod format

One of the challenges with the mod format is that there is not one definite mod file format but many different variants. Different programmers developed their own enhanced versions of the format that were backward compatible to some degree. As the hardware improved the format evolved to account for more hardware channels etc.

I am going to focus on the original format with the view of possibly extending it to cover the latter variants.

I have not found any fully authoritative source for the mod file formats but I have used the descriptions at the following links to understand the structure

• http://lclevy.free.fr/mo3/mod.txt
• https://www.aes.id.au/modformat.html
• http://sox.sourceforge.net/AudioFormats-11.html

Because the original format was developed for the Amiga computer it can be useful to have a look at the old hardware reference manual.

• http://ada.evergreen.edu/~tc_nik/files/AmigaHardRefManual.pdf

I am also using milky tracker to load and play different mod files to check that my interpretations match.

Reading files and using match

Before we can parse the file we need to read it from disk. The Rust standard library std::fs has a helper function fs:::open that reads the given file and returns a vector of u8.

The code for reading the file is below

    let file_data: Vec<u8> = match fs::read(file_name) {
        Err(why) => panic!("Cant open file {}: {}", file_name, why.description()),
        Ok(data) => data
    };

This contains a lot of Rust goodness that is worth spending some time on. The first thing to note is the match syntax used for handling the return value. This match syntax is effectively like C++ switch...case on steroids.

The value following the keyword match is the value being matched. In our case it is the return value from fs:::read that is being used. The curly braces enclose all the arms of the match. An arm consists of the pattern matcher on the left and handler on the right of the =>.

The match must be exhaustive so the arms must cover every possible value that the match could receive. In this case the return value from fs:::read is a Result enum which can have two variants; Err and Ok.

In Rust different instances of enum variants can have data attached to them. The variant Result::Ok has the actual data inside it and the variant Result::Err has the error. The enums with data is very nice concept that can help make the code more concise.

The match itself can be an expression. So the return value of the chosen arm can be assigned directly into a variable. In this case it gets assigned into file_data

The code above represents a fairly common sequence of operations so there is a helper method Result::expect that will return the value in Ok if that is the result and otherwise will panic and print an error message. Using the helper the above code condenses into;

fn read_mod_file(file_name: &str){
    let file_data: Vec<u8> = fs::read(file_name).expect( &format!(  "Cant open file {}", &file_name ) );
}

This is the code I will use but now I understand what it does can can use enumand match in other places.

Extracting data from Vec

With the data loaded into file_data we can start pulling out information from it. I will start with the easiest part; the name of the mod song. The first 20 bytes in all mod files store the song name. The following line copies that into a Rust string

    let song_name = String::from_utf8_lossy(&file_data[0..20]);

The code file_data[0..20] creates a slice of u8 data which is then passed into String::from_utf8_lossy which returns a String. The slice syntax uses the range operator a..b to specify which parts of the vector should be used for the slice.

Strings in Rust are utf8 which is why the conversion is required.

Next, I want to read and prepare the audio samples. Different versions of the mod format can have different numbers of audio samples but this number is not explicitly stated anywhere so the number of samples needs to be worked out indirectly. Many format variants have a format tag at file offset [1080...1084] The code will inspect this tag and use it to determine the number of samples in the file.

This kind of messy deduction needs to be put in its own function. So far the function only differentiates between files with the tag M.K. and those that dont but this is likely to grow as I encounter more files.

fn identify_num_samples(file_data: &Vec<u8>) -> u32 {
    let format_tag = String::from_utf8_lossy(&file_data[1080..1084]);
    match format_tag.as_ref() {
        "M.K." => 31,
        _ => 15
    }
}

Reading audio samples

Now that I know how many audio samples there are I can extract them. I have set up a structure that contain all the information about each sample ( the contents mirror the specs).

pub struct Sample {
    name: String,
    size: u32,
    volume: u8,
    fine_tune: u8,
    repeat_offset: u32,
    repeat_size: u32,
    samples: Vec<i8>,
}

I have created a matching sample constructor that takes a u8 array and uses it to create the sample structure.

impl Sample{
    fn new( sample_info : &[u8] ) -> Sample {
        let sample_name = String::from_utf8_lossy(&sample_info[0..22]);
        let sample_size: u32 = ((sample_info[23] as u32) + (sample_info[22] as u32) * 256) * 2;
        let fine_tune = sample_info[24];
        let volume = sample_info[25];

        let repeat_offset: u32 = (sample_info[27] as u32) + (sample_info[26] as u32) * 256;
        let repeat_size: u32 = (sample_info[29] as u32) + (sample_info[28] as u32) * 256;

        Sample {
            name: String::from(sample_name),
            size: sample_size,
            volume: volume,
            fine_tune: fine_tune,
            repeat_offset: repeat_offset,
            repeat_size: repeat_size,
            samples: Vec::new(),
        }
    }

The only special bits about this is the extraction of sample_size, repeat_offsetand repeat_size. They are 16 bit values in the file that are stored in a big endian format so we need to convert them into whatever endiannes we happen to be running on.

It is also worth noting that the argument to the constructor is of [u8] and not the entire file_data vector. This stops the code from accidentally reading from regions that it is not meant to.

To create all the samples I iterate over the relevant part over the file and call a sample constructor on each part.

    let mut samples: Vec<Sample> = Vec::new();
    let mut offset : usize = 20=;
    for _sample_num in 0..num_samples {
        samples.push(Sample::new( &file_data[ offset  .. ( offset + 30 ) as usize  ]));
        offset += 30;
    }

The 'magic' numbers in the above code are the length of each sample info block ( 30 bytes ) and the initial offset into the mod file ( 20 bytes ). I fully intend to convert them into consts/variables once I have a better understanding of the format.

Patterns and pattern tables

Mod files store the actual note data in patterns. Each pattern has 64 lines of note data that control sample playing and effects.

Mod files use pattern tables to control the order in which the patterns are played in. So a pattern table [ 0,1,2,1,2,] would mean the first play patterns 0,1,2 and the play 1 and 2 again.

The pattern table info is stored right after the sample data in the mode file. The first byte is the how many patterns long the song is and the second is the restart position. The following 128 bytes are the actual pattern data.

The code for reading the pattern table is

    let num_patterns: u8 = file_data[offset];
    let end_position: u8 = file_data[offset + 1];
    offset += 2;
    let pattern_table: Vec<u8> = file_data[offset..(offset + 128)].to_vec();
    offset += 128;

the above uses [u8].to_vec() to convert the pattern_table slice into a vector.

I still need to read the patterns and the actual sample data which will be the topic for my next post

I have put all the code on https://github.com/janiorca/articles/tree/master/mod_player