Monday, June 18, 2012

Multimedia Fusion and Blowfish

Multimedia Fusion is a great program made by a company called Clickteam. If you’ve ever heard of Klik ‘n’ Play, Multimedia Fusion (MMF) is the latest version of that. I started using Klik ‘n’ Play way back in the late 90s (probably 1998). I made some pretty bad games with it, which are still available for download on my web page. About a year later, I purchased my first copy of MMF and discovered that the makers of the software had their own web site and community. I never made any particularly good software with MMF, which is not to say MMF is incapable of good software. It is. In fact, several people I know have made successful commercial ventures with games they made in MMF. I really found my niche in the community when I started learning C++ the summer before my senior year in high school. Suddenly, I could make extensions for MMF, adding new functionality to the program. I may not have been much of a game maker, but I wasn’t too bad with making useful extensions. Eventually, I developed a relationship with some members of the company. In fact, they’ve sent me a few free copies of their software over the years. Alas, over time, I grew apart from MMF and the community. Throughout college, I focussed more and more on traditional programming paradigms and less on MMF. I preferred the more flowing structure of languages like C++ than the event list structure of MMF.

Eventually, I lost all contact with the community. This is something I regret quite a bit. Occasionally, I would get e-mails from people asking about my extensions but not too often. Back in 2008, I was made aware of a bug in one of my extensions, the Blowfish encryption object. This is the one extension that I had actually sold to Clickteam, so when I decided that I would try to fix the bug, I had to get a copy of the source from them. I actually found and fixed the bug. It turns out that the Blowfish algorithm expects data in big-endian form (I’ll discuss this more later), so I just needed to reverse the byte ordering before and after encryption. I did this, but then I got an even bigger idea in my head. It seems silly that each object that wants to use encryption should have to implement the algorithm itself. What if there were a generalized encryption system where any object that wanted to encrypt data could simply be passed the information of an encryption object and could then make use of the encryption object’s own algorithms? Well, I set about working on that model and got in over my head. After a while, I gave up and never returned the fixed source code to Clickteam. I was in a time in my life where motivation is difficult to find. Years passed. I’m finally now trying to dig myself out of this rut. In fact, this blog is one of the ways I’m trying to do it. I figure that the more I tell people what I’m doing, the more likely I am to follow through with it. Granted, no one reads this blog, but maybe, some day, someone will. I consider not correcting the bug in the Blowfish object to be a significant failing on my part, so finally fixing it is of great importance to my sense of self-wroth.

The bug in question, as mentioned earlier, had to do with the byte ordering of data sent to the encryption/decryption functions. If you don’t plan to use any encrypted data with any other implementations of Blowfish, this isn’t actually a problem, but it is likely that you’d want to do that. When passing data (stored as an array of characters) to the Blowfish functions, I would just typecast addresses into the array as pointers to unsigned longs. In little-endian formats, this puts the first character as the lowest byte of the newly formed long. It seems that Blowfish implementations prefer that this first character be in the highest byte of the long. I’m not entirely sure if this means that Blowfish implementations prefer big-endianness or if it’s simply that they convert the array of characters using bit-shifts and ors rather than simple typecasts. I wasn’t satisfied, however, with just fixing this bug. I wanted to add some different cipher block modes. In the past, I had only used the electronic codebook mode (ECB) because I simply wasn’t aware of the other modes in use. At first, I was only going to add cipher-block chaining mode (CBC), but after further investigation, I realized that cipher feedback mode (CFB) and output feedback mode (OFB) were relatively simple to implement. I decided to add all three new modes to the object. I’m not sure what the relative advantages and disadvantages are of each mode, but since they were so simple to add, I decided to just add them and let the user sort it out.

I also wanted to rewrite the file-handling function. In my younger days, I seemed to have a strange dislike of keeping files open, so I would read entire files into memory and then process the data. Once done, I would reopen the file and write the new data. For small files, this is fine, but for large files, this can be problematic. In rewriting the function, I had to open the file in both read and write modes. I had never used a file in this manner before and was unaware of one of the quirks involved. In my first tests, I was able to read and write my first block of data, but I was unable to read subsequent blocks. My first test took a little 1.82 MB file and turned it into a gigantic 64 MB file because I would keep reading and writing the same block of data over and over. After some research, I discovered than when switching between using fread and fwrite, you need to have a call to fseek. Before writing data back to the file, I would call fseek to jump back to the start of the block, but since the write operation placed me at the location in the file of the next block of data to read, I had no reason to call fseek again. Because an fseek call is required before switching back to fread, I had to add a dummy call to fseek that doesn’t change my position in the file. After fixing that problem, my updated file handling worked perfectly.

I took me nearly a week to make and debug all of the changes that I wanted, but I feel it was worth it to remove this blemish from my psyche. I’ve returned the updated object to Clickteam to see what they think of the update object.

15 comments:

  1. Hi Matt!

    I still miss our fun in vchat!
    Happy to see you are alive and well!

    Later,
    Jeff V

    ReplyDelete
  2. This comment has been removed by the author.

    ReplyDelete
  3. Hi Matt. I'm struggling to get my head around what filtering I need to put in place to decrypt MMF blowsfish encrypted files without MMF. I tried flipping the endianness of every 8 bytes, but the result was garbled rubbish. steve (at) atulos dot com.

    ReplyDelete
    Replies
    1. If you're using Reverse Bytes mode, you shouldn't need to do anything special. Otherwise, I think you'll need to reverse the order of every 4 bytes.

      If you had the encrypted bytes:
      byte0 byte1 byte2 byte3 byte4 byte5 byte6 byte7
      you would reorder them as
      byte3 byte2 byte1 byte0 byte7 byte6 byte5 byte4
      and then decrypt them. Finally, reorder them back to their original order.

      Delete
    2. Thanks. When you said "unsigned long", I assumed 8 bytes. Silly me. I flipped every four byte sequence, ran it through the Blowfish/ECB/NoPadding cipher then flipped the output. It worked!

      Delete
    3. What does Enable Filtering of Characters with Codes 0 to 31 mean?

      Delete
  4. It's been a while since I've looked at the object, but if I recall correctly, filtering is done when strings are encrypted (or when the Filter format is selected for output with EncryptStringAdv$). Filtering takes any characters in the result that are set to be filtered (% and the NULL character by default) and converts them to %xx, where xx are two hexadecimal digits representing the code of the character. When strings are decrypted (or when the Filter format is selected for input with DecryptStringAdv$), those %xx things are converted back into their correct characters before any decryption is done.

    The % and NULL characters are the only ones that NEED to be filtered because the NULL character is used to mark the end of a string so not filtering it could cause your encrypted string to be truncated and because the % character is used in the filtering process. You can optionally filter other characters as well. The "Characters with Codes 0 to 31" are a special set of characters that are generally used for formatting (like line breaks) and don't tend to have actual picture representation. Some people may prefer to filter all of these out, so I added that option. You can safely ignore it that option if you'd like.

    ReplyDelete
  5. Hi Matt,

    I have to be able to replicate Blowfish Object encryption/decription for legacy purposes. I've been trying to crack it for 2 days now and I really need your help.

    http://play.golang.org/p/sKrXmdoDSg

    Here is my code. It always returns garbage, no matter what I try to do.

    Here is the one without byte swapping with properly encrypted Blowfish file (binary).
    http://play.golang.org/p/-AMPNp0fzy
    Works like charm.

    Please, can you assist me in all the things I need to replicate in order to be able to encrypt/decrypt Blowfish Object compatible files?

    ReplyDelete
    Replies
    1. Ok, this code seems to work when I encrypt a file wdata.dat with BlueFish Object that contains a single word: foo.

      http://play.golang.org/p/hrBg1BEoNb

      But as soon as I have multiple words or add newlines, spaces or special characters, I get garbage.

      Any ideas why?

      I disabled 'Filter characters 0 to 31' and both text boxes are empty.

      Delete
    2. http://gyazo.com/0a024f485c150359423848ce99eec0c2

      This:
      http://gyazo.com/08583a5d0f90045c37d64005d4ce0c5f

      Becomes:
      http://gyazo.com/dc90c417c0a46e682c0d09508d02c410

      Delete
    3. It looks like you're using an old version of the Blowfish object. The updated version supports a Reverse Bytes mode that will make the object's encryption compatible with other implementations so you won't need to bother with reversing bytes in your Go code. Unfortunately, you can't use Reverse Bytes mode with data encrypted with the older versions of the Blowfish object (if that's a concern, you can just disable the Reverse Bytes mode).

      A few notes:
      -Filtering is only done on strings. It will not be done on files so your filtering options are irrelevant when dealing with files.
      -You have disabled file padding. This means that if the size of your file is not a multiple of 8, the last few bytes will not be encrypted. For example, your "foo" file is only 3 bytes so no encryption would be done on it.

      I'm not familiar with Go and the sources you posted wouldn't run for me because it seems that the site had trouble locating Blowfish.

      Delete
  6. Yes, the site is a 'sandbox', most of real world code can't run on it. I just use it becaues its 1 stroke away from Vim.

    How do I get the updated version of Blowfish Object?

    Also, I am using padding. Without padding, the len(b)%8 check panics. So when I encrypt, and decrypt with Blowfish Object, there are empty spaces left in the file on each line.

    ReplyDelete
    Replies
    1. I'm not really sure where to get the latest version. The Blowfish object was distributed by Clickteam as part of one of the bonus packs. When I updated the object, I just sent it to them. I assume they updated it in the bonus pack, but I never looked into it. Try downloading and reinstalling the bonus packs.

      I looked into Go a bit more and it looks like it only encrypts/decrypts the first 8 bytes that you give it. You would have to loop through each 8-byte chunk to encrypt/decrypt more than 8 bytes. It looks like you may be able to make use of some encryption modes but I'm not sure how that works in Go. The old version of the Blowfish object used the ECB encryption mode but the newer version of the object has support for better encryption modes.

      Also, I couldn't find any documentation that discusses how files are read in Go. I'm assuming they read in binary mode, but if they're reading in text mode, this could cause some problems in files containing new lines.

      Delete
    2. Ahhh, super silly me. I didn't read the documentation well enough, it does actually just do first 8 bytes. So I feed it 8 bytes at a time and it works perfectly now.
      http://play.golang.org/p/7QWMKyK0Ej
      http://gyazo.com/ecf156eca4052277c7fe9f89bbe02db8

      Thank you so much Matt, you are awesome :)

      Delete