The term is ending very soon. I’ll be continuing on the project as part-time till early next year and we do have few other directions potentially can go.

Iteration Blending

The blending based on iteration is fully functional. If the configuration requires 10 iterations in total to dream on one frame, the blending algorithm will dream 1 time for style A, then 9 times for style B. Next frame, the logic will apply 2 times A style dreaming and 8 times style B dreaming. This pattern continues until the style fully moved from style B to A. Right now a recursion algorithm is applied to implement this idea. Performance is not that good where roughly takes 10 seconds to finish a 10 iteration frame (2 passes). The result is much improved in regard to the ghost effect of the alpha blending technique.

Refactoring & Python

Some heavy refactoring is taking place recently and few interesting observations are done. Python does offer fairly strong support for scientific researching with easy API for image processing, video analysis, high dimensional array. However, so far my experience with it is still bad. The style of the programming is more like a quick mockup to implement an idea rather than seriously taking care of the integrity of the code.

One example can be the epic indentation logic to decide code block. For a giant nested control statement like if, for etc. It is quite likely the ending become chaotic where statements are floating around and no clue who belongs to who. For me, this is purely a design flaw that I gain nothing while only introducing trouble later to maintain. And my conclusion is based on that a good design is able to produce robust code logic not only for now but for future reference while simplified syntax is ONLY a bonus if the earlier foundation can be laid properly.

for x in range(0, n):
    # Do some stuff here...
    for y in range(0, n):
        # Do other stuff here...
        if foo is None:
            # Keep doing stuff...
    # Indentation here will be confusing who it belongs to.<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

Variable Parameter Passing

In Python, *args and **kwargs can be used to pass an indefinite number of parameters to a function in the function definition. This feature is also common for other programming languages. It helps some scenarios where the input of the function cannot be predetermined but still following a pattern. For example, a function converts a string user input to a book title style with uppercase staring characters. To simplify the logic, a user can separate an arbitrary number of strings and pass as parameters instead of containing them in a collection object, say array or list. Specifically, this logic is introduced in our dreaming code:

dreamed_frame = deep_dream(
            net,
            input_frame,
            image_type=image_type,
            verbose=verbose,
            iter_n=int(key_frames[i].iteration.first),
            octave_n=int(key_frames[i].octave.first),
            octave_scale=octave_scale,
            step_size=stepsize,
            jitter=jitter,
            guide_image=guide_image,
            end=end_param,
            )


def deep_dream(
    net,
    base_img,
    image_type,
    iter_n=10,
    octave_n=4,
    octave_scale=1.4,
    end='inception_4c/output',
    clip=True,
    verbose=1,
    guide_image=None,
    **step_params
):<span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>

    # Do stuff...

        for i in xrange(iter_n):
            make_step(net, end=end, clip=clip, guide_features=guide_features,
                      **step_params)

    # Do stuff...

For the code above, the “deep_dream” function is called with procedure “make_step” inside. In line 27, the “step_params” is defined can be multiple parameters and directly passed to the “make_step” procedure in line. The actual parameter passing is in line 12 where the “end_param”.

One good thing about this is it makes calling function able to separate the immediate parameters with the ones will be passed later to other nested functions. For example, in a real-life cookie making scenario, Annie is dedicated to cookie making while Joe is dedicated to plate cleaning. If I want a plate of cookies, the traditional way is asking Annie:

“Hi, Annie, would you make a plate of cookie for me with powder A, salt B, blueberry C, plate D, kitchen cleaner F, and paper towel E?”

Annie accept the task then ask:

 “Hi, Joe, would you prepare me a container with plate D, kitchen cleaner F, and paper towel E?”.

Later on, Annie will be able to make the cookie with the plate Joe prepared. To use the variable parameter, I will ask:

“Hi, Annie, would you make a plate of cookie for me with powder A, salt B, blueberry C, and stuff for Joe?”

The stuff for Joe will contain “plate D, kitchen cleaner F, and paper towel E” which Annie is not sure about due to her dedication to cookie making. Then, Annie can acquire the plate by asking:

 “Hi, Joe, would you prepare me a container with stuff I got from Wei?”.

Essentially, the same scenario can happen in the programming world. The variable parameter allows programmer passing parameters without knowing the actual content of them which are encapsulated inside the *args pointer. This reduces the risk that mistakenly using the parameters since the validation will be done in the nested functions, Joe.

However, there are few drawbacks too with this idea. First, the programmer will be hidden from the passing parameters. If something is wrong and the programmer is forced to dig into the nested function, Joe, it is difficult to debug since which parameter is passed is unknown and all appended to that pointer. Second, the nested logic is not recursive. Say I pass the information, ingredient, plate, and water switch to Annie, Annie pass plate and water switch to Joe, Joe cannot pass the same thing, plate, water switch to plumber Smith but have to trim the plate and only pass the water to Smith. Although the trim can be done in advance with a nested pointer at the beginning, it is just too much trouble.

In conclusion, this variable parameter is more likely have no special application than the typical scenario other programming language offers. Say, an undetermined number of user input needs to be handled. And from my past experience, this situation is not that common. The particular case we have in the deep dreaming algorithm seems not really on point to practice the variable parameter idea since the input is fixed and the drawback of such implementation is more significant.

Named Tuple

In mathematics a tuple is a finite ordered list (sequence) of elements. An n-tuple is a sequence (or ordered list) of n elements, where n is a non-negative integer. There is only one 0-tuple, an empty sequence. An n-tuple is defined inductively using the construction of an ordered pair. — Wikipedia

One example is a Point(x, y) can be a tuple consist of x and y value. Python example can be found as follow:

# Declare and instantiate tuple.
tup1 = ('physics', 'chemistry', 1997, 2000);
tup2 = (1, 2, 3, 4, 5);
tup3 = "a", "b", "c", "d";

# Access tuple.
print "tup1[0]: ", tup1[0];
print "tup2[1:5]: ", tup2[1:5];

This feature can be very helpful for some light weighted object say a point, instead of creating your own object, just use a tuple. However, one problem is the naming is less convenient since p[0] has little connection with x coordinate of a point. The solution is named tuple:

from collections import namedtuple
Point = namedtuple('Point', 'x y')
pt1 = Point(1.0, 5.0)
pt2 = Point(2.5, 1.5)

# Compute displacement from two points.
from math import sqrt
line_length = sqrt((pt1.x - pt2.x) ** 2 + (pt1.y - pt2.y) ** 2)

One thing about named tuple is that it is likely always better than the nameless tuple. Unless it is in a context impossible to name the tuple. To utilize this feature, now all the keyframes are built in named tuple so frame[i].iteration.first and frame[i].iteration.second will be the two weights for the iteration blending mentioned at the very beginning.

In contrast, C++ seems to have the similar idea to encapsulate in a light-weighted object while Java, as expected, does not have. It is more like a culture thing that Java tends to be robust while rarely predict the specific user scenario. So Java users are expected to implement their own light-weighted object for the tuple, at least pair (xxx.first & xxx.second).

The painting project made some progress recently. It now supports linear keyframe interpolation on iteration, octave, layer, and model. The video clip can relatively transit from one style to another. However, temporal coherence is still an issue to be solved. The entire creative team gathered recently which lead to another deeper solution potentially to help all the existing issues.

Deep Dream Basics

The essential idea how deep dream apply the style to the image is quite simple. The algorithm run over an image array and generate RGB delta of each pixel based on the original value. Then, this value is recursively enhanced by apply same technique again and again. Essentially, the out equals to original pixel value plus deep dream delta.

Image Blending Issue

Based on above information, instead of using alpha blending on two generated images, which can lead to significant “ghost” artifacts, we can simple dream on different style with various iterations. For example, instead of alpha blend two images from style A, 70% alpha, 10 iterations to B, 30% alpha, 10 iterations, we can simply apply A style iteration 7 times, then B style iteration 3 times recursively on the same image.

The only issue about this new method is the order of the iterations. They can be mixed ABABA, BAAAA or BBBAA, AAAAA or AAAAA, AABBB. The results from these methods will be different. An evaluation of the results will be required after the code is done.

Temporal Coherence

This issue has already been addressed again and again. Previously, by using optical flow to direct the dreaming is regarded to be one of the best solutions. However, detailed implementation is unknown. Now, we can finally apply the vector results from the optical flow to the deep dream delta result to redirect the result.

One concern is we still want the dream progress based on time. For example, if a sun moving from left to right, the expected results say a flam texture is gradually evolving while the sun is moving instead of a fixed texture on the location where the sun is. In that way, one evaluation must be done to test will slight different from the source image make a huge change for the final dreamed result. Only if it is negative, we might further utilize the optical flow method mentioned.

AI Avatar Meeting

I also attended the other meeting about an AI avatar program, an immersive talk bot. Few challenges are addressed there including:

• Implementing more XML command to adjust talking environment light, avatar body movement etc.
• Build a queue system to handle multiple gestures if they overlap together
• Research potential solution move the project to Unity

In the future, I may also help out the team to do some programming about above issues.

It didn’t take a long time to setup my dev environment. After the remote connection is done, which I have to do since the lab is running on some high-end CUDA machines, few well-labeled programs are gathered my own folder. A quick test is run against a sample video to separate it into frames, apply deep dreaming on the frames, then assemble the frames with sounds back to a dreamed video. The process takes few hours for ~1000 frames and the result is fine.

Iteration & Octave Interpolation

As mentioned in the previous post, iteration and octave are added by a new parameter pointing to a CSV file as follow:

frame,iteration,octave,layer
33,15,5,Layer
66,5,30,Layer
99,30,10,Layer

These values then can be interpolated linearly or by other models to generate a tuple list with the corresponding iteration, octave, layer (not implemented yet). Every two rows will be processed in one iteration which marks the start and the end of one keyframe section. One trick here is if the first keyframe is not starting from frame 1, the assumption is made that anything between frame 1 and this keyframe will have same iteration, octave, and layer. Same thing applies to the last keyframe and the last frame. The code sample is as follow:

# Must add one extra since frame start from 1 rather than 0
keyFrames = [(iterations, octaves)] * (int(totalFrame) + 1)
if key_frames is not None:
    with key_frames as csvfile:

        # Get first line to decide if the first frame is defined
        reader = csv.reader(csvfile, delimiter=',', quotechar='\'')
        reader = csv.reader(csvfile, delimiter=',', quotechar='\'')
        next(reader, None) # Skip the header
        firstLine = next(reader, None)
        if firstLine[0] != '1':
            previousRow = firstLine
            previousRow[0] = '1'
        else:
            previousRow = ''

        # Rewind the reader and read line by line
        csvfile.seek(0)
        next(reader, None) # Skip the header
        for row in reader:
            if previousRow != '':
                interpolate(keyFrames, previousRow, row, 'Linear')
            previousRow = row

        # Check last line and end interpolation properly
        if row[0] != str(totalFrame):
            lastRow = row[:]
            lastRow[0] = str(totalFrame)
            interpolate(keyFrames, row, lastRow, 'Linear')

After these sections are prepared, interpolation function is called. Currently, only the simple linear model is implemented where more advanced ones can be introduced in the future. One thing should keep in mind is that frame always starts from 1 rather than 0. The code sample is as follow:

def interpolate(
    keyFrames,
    previousRow,
    currentRow,
    method
):
    iterationFragment = (float(currentRow[1]) - float(previousRow[1])) /\
                        (float(currentRow[0]) - float(previousRow[0]))
    octaveFragment = (float(currentRow[2]) - float(previousRow[2])) /\
                     (float(currentRow[0]) - float(previousRow[0]))
    for i in range(0, int(currentRow[0]) - int(previousRow[0]) + 1):
        iteration = str(int(float(previousRow[1]) + i * iterationFragment))
        octave = str(int(float(previousRow[2]) + i * octaveFragment))
        keyFrames[int(previousRow[0]) + i] = (iteration, octave)

About Python

Apparently, I’m a Python hater 😀 One advice I’m always giving is do not start your programming career with Python. Reseason behind this is Python is such a different language compares to C-like style, fully adapt to Python will cause people difficult to move to other languages, i.e. I have friends tend to keep forgetting a variable need a type and how type casting works when moving from Python to others.

Other things are just pure basics, indentation rather than {}, don’t need to declare a variable before using it, no strict type. It literally dodged every single criterion where I think a good programming language should have. However, its “simplicity” and some well-supported science libraries made it gaining popularity in the research world. Nevertheless, one thing I learned from the past is if you are seriously thinking to work with one language style, you’d better choose the one pleasing you when writing.