8 processors but still waiting

Our new build system has lots of ram and a multicore hyperthreaded cpu: 4*2 processors. It is just unbelievable how responsive this Vista-64 machine is.

We are currently making some changes to the build process of WatchFTP, creating separate installers for each supported language. Previously, we created just one installer with a multilingual executable (thank you Sisulizer for a great localization product). The multilingual approach worked fine most of the time, however when a German customer living in France wants to run the English version of WatchFTP on his Spanish-language operating system it was just not possible. The operating system would select the language to display, not much you could do about it.

Future WatchFTP versions will have the desired display language build-in. Sisulizer will create language specific (German, Spanish etc) copies of the original (English) executables. The display language will only depend on the installer you download from our websites.

Steps in our build process

1. Build a "release" version of WatchFTP
   (Microsoft Visual Studio 2008 - C++ by the way)
2. Run Sisulizer to create the multilingual executables
3. Protect the core components (2 executables)
4. Code-sign the EXE and DLLs
5. Run Inno Setup to create the installer
6. Code-sign the setup file
7. Test the setup
   When we find issues, fix them and go back to step 1.
8. When all is OK, a lot of other tasks like uploading to our website etc

The most time consuming step (several minutes) was step 3 (software protection). Not really a big deal, we don't release new versions that often. However, step 7 (test setup) can cause a lot of rebuilds (start at step 1) and the delays for software protection become very annoying.

It gets Worse!!

When we implement separate installers for the different languages we support, we will have to do step 3 (protect) for each language (currently German, Spanish and English. Soon we will have a French version as well).
This will make our already long build process 4 times longer - totally unacceptable.

Our great new computer doesn't help at all because all those steps need to be sequential. We can not code-sign (step 4) before the protection (step 3) is completed. Also, the protection phase is done with a single-threaded program, even if the computer has 1000 cpus, it will only use one of them.

How we solved it

Obviously, the protection step (nr 4) needs to be paralleled. We can launch the protector as a separate process for each of the languages. A first attempt (WRONG!!) in our build batch script:

....
ECHO ==STEP 2 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 3 Starts - Software Protection==>>C:\Temp\buildlog.txt
Set LANGUAGE=ENGLISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=SPANISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=GERMAN
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=FRENCH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
ECHO ==STEP 3 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 4 Starts - Code Signing protected Executables ==>>C:\Temp\buildlog.txt

(note: the scripts shown in this post are very simplified without error checking etcetera).
The protect.bat file is called and based on the LANGUAGE variable it will protect a specific language version of WatchFTP. The beauty of the Start command is that it will launch a separate console (command prompt) to run the protect.bat script. In the example above 4 language versions are protected simultaneously, each using their own CPU. Obviously, because of disk usage, there is some slowdown - it won't complete 4 times as fast.

A small problem...

In case you didn't realize, the above script is sort of wrong: it continues running while the 4 "Protect.bat" files are not finished yet. So it gladly goes on to Step 4 (code signing), but there isn't anything to code-sign yet. We need to wait for the 4 protect.bat processes to complete before we can continue with Step 4.

And an easy fix

We have a small command prompt program called GdPUtil (Free Download). Please read the description of the -join parameter.

We added the following lines at the end of protect.bat so it signals to the main script it is ready:

....
"C:\Bin\GdPUtil.exe" -join protect%LANGUAGE% 1000
EXIT

And changed the original build script to

....
ECHO ==STEP 2 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 3 Starts - Software Protection==>>C:\Temp\buildlog.txt
Set LANGUAGE=ENGLISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=SPANISH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=GERMAN
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"
Set LANGUAGE=FRENCH
Start "Protecting %LANGUAGE%" "E:\Build\Protect.bat"

ECHO ==STEP 3 waiting for protection to complete ==>>C:\Temp\buildlog.txt

"C:\Bin\GdPUtil.exe" -join protectENGLISH 1000
"C:\Bin\GdPUtil.exe" -join protectSPANISH 1000
"C:\Bin\GdPUtil.exe" -join protectGERMAN 1000
"C:\Bin\GdPUtil.exe" -join protectFRENCH 1000

ECHO ==STEP 3 Complete==>>C:\Temp\buildlog.txt
ECHO ==STEP 4 Starts - Code Signing protected Executables ==>>C:\Temp\buildlog.txt

Now our script works like a champ! Even though it has to do 4 times as much work, it is only slightly slower than it used to be.

Why does this matter to you?

If you have long running scripts, have a good look if they can be broken up like we did in our build script. For example, are you uploading a file to multiple FTP servers in your script? If you have enough upload bandwidth but it is the FTP servers causing the slowdown, this technique can be very helpful.

SkunkWorks - New Program, New Name, Contest, Testers.....

Hello,

We are currently working on a new program. Functionality is going great so far but we have a strange problem ;-) More on that further down below...

The Program

So far our programs WatchDirectory and WatchFTP concentrated on doing things automagically. This new program is different.
It allows you to select files (in Windows Explorer, for example), add these files to a "basket" (an icon floating above explorer) and then perform "actions" on those baskets. A few actions we are currently implementing:

• COPY - copy the files dropped into the basked to a pre-configured directory.
• MOVE - same as copy, but move the files instead.
• EMAIL - email the files to a pre-configured email address.
• FTP - upload the files to an FTP server.
• BATCH (you already knew this was coming, I guess ;-) - Run a script for each of those files - program your own action.
• YourIdea - we are always interested to hear from you.

The Name

So... if you read the above... any idea for a good name? There are a few important features the name must have:

• English - our biggest market.
• Web - the www.TheNameYouThinkOf.com name must be available.
• Attractive/Mysterious - if the name wows people who see it, making them curious, that would be great!
• Not offensive - need I explain this?
• File - it would be great if it has the word "file" in its name.
• Short - not as important as the above, but shorter names are easier to type (url in webbrowsers).

We already thought of several names (all very boring), do you know a good name? A good program name can make the difference between success or failure for a new program, so if you suggest the "winning name" to us we will compensate you for it. Beside a free copy of the program and our everlasting gratitude - a $1000 reward or 50% of the first month sales of this program - you choose!

Send an email to gert@watchdirectory.net, use the subject "SKUNKWORKS - Name". Tell us the program name and why you think it is a great name to use.

Beta Testers

It may take a month, maybe 2 months before we have the first beta ready, but we need beta testers for this new program. Beta testers receive a free copy when it is released, obviously. Interested?

Send an email to gert@watchdirectory.net, use the subject "SKUNKWORKS - Beta".

Update - July 22, 2009

We just bought the domain www.FileHalla.com (thank you Peter Wester) - any Vikings reading this ;-)
If you have a better idea, the contest is still open, probably until the end of the month.

Update - July 25, 2009

We just bought the domain www.FileViking.com (thank you Al Harberg). He didn't actually suggest this name but warned me about negative feelings the name Valhalla/Walhalla might cause.

Am I sure about FileViking? I think some ancestors of mine were slaughtered by Vikings... It has been some time ago, though...

Tweet your files with curl

This post shows how you can let WatchDirectory or WatchFTP post a new twitter message when they detect new files. Excuse me if I get some of the twitter terminology wrong - I grew up when computers were something you see in movies ;-)

Curl

This page shows that posting a message to twitter is as simple as

curl -u username:password -d status="your message here" http://twitter.com/statuses/update.json

So, we need Curl to post. I used the non-ssl version while testing. Make sure to download a Windows version from their download page.

Batch File

If you want to "tweet" when a new file is detected, create a new task and select the Run a Batch Script task (for WatchDirectory) or Run a Batch Script Action (for WatchFTP).

Let it run the following batch file (may not show correctly in all blog-readers):

@echo off
SET CURL=E:\bin\curl-7.19.5\curl.exe
SET TWUSER=TheTwitterUser
SET TWPASS=TheTwitterPassword

rem if we are called by WatchFTP, copy its variables to WD variables
rem so this script is compatible with both programs

IF "%WF_REASON%" NEQ "" (
  SET WD_REASON=%WF_REASON%
  SET WD_FILE=%WF_FILE%
)


GOTO :%WD_REASON%

:FILENEW
SET MSG="New file: %WD_FILE%"
GOTO :SendIt
:FILEDEL
SET MSG="Deleted file: %WD_FILE%"
GOTO :SendIt
:FILECHNG
SET MSG="Changed file: %WD_FILE%"
GOTO :SendIt
:FILEREN
SET MSG="Renamed file: %WD_OFILE% to %WD_FILE%"
GOTO :SendIt
:DIRNEW
SET MSG="New directory: %WD_FILE%"
GOTO :SendIt
:DIRDEL
SET MSG="Deleted directory: %WD_FILE%"
GOTO :SendIt
:DIRREN
SET MSG="Renamed directory: %WD_OFILE% to %WD_FILE%"
GOTO :SendIt


:SendIt
"%CURL%" -u %TWUSER%:%TWPASS% -d status=%MSG% http://twitter.com/statuses/update.json

Make sure to properly change the first few lines (where did you install curl, what is your twitter userid and password).

Twitter Limits

A tweet is a short message (I believe limited to about 140 characters). If your directory and filenames are long, you may need to shorten the messages a bit. For example, change %WD_FILE% to %WD_FILE_N% everywhere in the script so it will only tweet the filename (without the directory path).

Twitter also has some other limits you may hit if you call it too often.

Blogging your Files?

This post shows you how.

WatchFTP 2.2 beta

A beta version for WF 2.2 is available on our forum.

New in this beta release is that you can enter all kind of "timing" related settings for each task. Among other things, this gives you easier access to all the "secret settings" described in this blogpost.

This beta does not have the new features translated to German and Spanish yet - the new items will show in English.

To install (and keep your current settings and tasks):
* Do NOT uninstall your current WF version
* Install the beta and use all proposed settings of the installer (same directories)

WatchDirectory 4.5.9 released

We just released a new WatchDirectory version - another free upgrade for all WatchDirectory version 4 customers.

Changes in this release

• Fixed: The tip window sometimes caused the WD Control Center to crash at startup
• New: The Auto Copy plugin can now MOVE files. When you move files on the same disk, this is faster than copy followed by delete.
• New: The Folder Size plugin writes a timestamp and filename of deleted/moved files to <Task Directory>\<Task Name>\removedfiles_YYYYMMDD.csv. These files are automatically removed when they are a month old.
• New: The Folder Size plugin can check for several new "events"
• Fixed: The "Last Message" column of the WatchDirectory Control Center was not always updating properly
• Fixed: The Auto UNZIP task now supports far larger files (tested with a 2GB zip containing an 8GB file)
• Fixed: The Auto ZIP task now supports far larger files (tested up to 8GB)
• Fixed: The Auto Copy option to remove directories when they become empty in some cases also removed the monitored directory.
• New: The thumbnail plugin can now be used as a SubTask in the SubTask plugin
• New: The thumbnail plugin has a new option to prevent overwriting existing thumbnails
• New: You can set the "color" of a task. If you have a lot of tasks, this allows you to organize them in different "groups". Give a task one of 6 colors using the "right click" context menu. You can sort the tasks by color by clicking the header in the WD Control Center. You can select all tasks having the same color using the new "Colors Toolbar".
• New: When you let the email plugin combine emails, you can now set the separator between emails yourself.
• New: Use CTRL+F (or View -> Find) to find and select tasks.
• Fix: The Automatically Start any Program plugin did not properly set the environment variable WD_REASON when a new directory was detected.

How to install

Stop all your WD tasks, stop the WD Control Center. Now install the new version in the same location as your previous installed version (overwrite it).
This will keep all your settings and tasks intact.

Download

Download Here (English) or Here (German).

July Special - Free ShowSize license

When you buy 2 licenses of WatchDirectory or 2 licenses of WatchFTP during July 2009, you can get a free license of ShowSize, normally sold at $49.95.

To get your free copy of ShowSize, forward your WatchDirectory or WatchFTP order confirmation to support@watchdirectory.net. Please allow a few days to process your request.

About ShowSize

You already know that Windows Explorer does not show folder sizes in its list. ShowSize can do that and much more. See how your disk space is distributed in various folders and file types, and so on. ShowSize is the next generation software that can process terabytes of disk space and millions of files. Using ShowSize, you can also list or print folder contents in ways not possible with Windows Explorer.