Dr Dave

Multitouch Surface support for XNA-based applications is provided via the Microsoft.Surface.Core.Manipulations namespace in an event-driven model. I was interested in investigating general multitouch processing within a game-loop model, i.e. not involving events, and thought it would be instructuve to develop my own class to process manipulations in this way.

In a similar manner to the Affine2DManipulationProcessor, I wanted to support arbitrary combinations of translation, rotation, and scale. Translation is trivial, since it is simply a matter of tracking the changes in each manipulation point between calls and taking the average.

Rotation and scale is a little more tricky, since each of these values are relative to a centre-point, or manipulation origin. Rotation is calculated from the average angle between each point and the origin, and scale is calculated from the average distance. Unlike a mouse cursor, manipulation points come and go, so the key is only to track changes to those points which persist between calls, and then update the positions, origin, distance and angles for the next call.

In order to generalise manipulation points from multiple sources such as mouse, gamepad, Surface Contacts and .NET 4.0 Touch, I created my own Manipulation structure to abstract properties such as ID and position. My input processing can then build a collection of these objects from any relevant source(s) and pass them to my manipulation processor as part of the Update call as follows:

myManipulationProcessor.ProcessManipulators(myManipulators);

The cumulative transforms for translation, rotation and scale (according to the supported manipulation types specified) are then immediately available as properties on the manipulation processor.

In order to test the solution, I wrote a small harness to visualise the manipulation points, and their effect on a reference cross-hair. While I also added support for a mouse (right-click to add/remove points, left-drag to move points), multi-touch hardware is required to test scenarios when multiple points are added, moved, or removed simultaneusly. A screenshot is shown in Figure 1.

Figure 1. Multitouch manipulation with averaged manipulation origin.

One of the complexities of working with multiple manipulation points is deciding how to determine the manipulation origin. One option is to simply take the average position of each point, as shown in Figure 1. Another option is to introduce a speicific "pivot point" as the origin and use this to calculate scale and rotation. This pivot point can either be fixed, or tracking the object being manipulated. The latter two options are shown in Figures 2 and 3.

Figure 2. Multitouch manipulation with fixed pivot point.

Figure 3. Multitouch manipulation with pivot point centered on manipulation object.

The solution is demonstrated in the following video. Each approach for determining the manipulation origin is illustrated using translation, rotation, and scaling manipulations, initially in isolation and then in combination.

Video 1. Multitouch game loop manipulation processing.

In Part 1 I discussed an simple shader which generated autostereograms from depth maps. The next step was to make the application interactive, so that instead of generating an autostereogram from a static depth-map, an interactive 3D model could be used to generate dynamic autostereograms.

To create a depth-map from a model, I added a pixel shader to return the depth value. A similar example can be found in How To: Create a Depth Texture. In order to obtain a sensible range of depth values, I need to calculate the bounds of the model using bounding spheres. I can then set the near and far planes in my projection matrix to correspond to the closest and furthest points in the model respectively. The depth texture can be extracted by rendering to a render target. It can then be passed as a shader parameter to allow autostereograms to be rendered in real time as the model is manipulated.

One of the main issues with animating an autostereogram is that manipulating the model results in changes across areas of the image which do not correspond to the location of the model itself.¹ One way around this distracting side-effect is to use a SIRDS, or Single Image Random Dot Stereogram, and alter the pattern each frame in which the model is manipulated.

Instead of passing a pre-defined pattern texture, I generate a random texture on the CPU as follows:

private Texture2D CreateStaticMap(int resolution)
{
    Random random = new Random();
    Color[] colors = new Color[resolution * resolution];
    for (int x = 0; x < resolution; x++)
        for (int y = 0; y < resolution; y++)
            colors[x + y * resolution] = new Color(new Vector3((float)random.NextDouble()));

    Texture2D texture = new Texture2D(GraphicsDevice, resolution, resolution, 1, TextureUsage.None, SurfaceFormat.Color);
    texture.SetData(colors);
    return texture;
}

The shader then tints the black & white image to give the result shown below in Figure 1.

Figure 1. SIRDS for animation

¹This is exacerbated by my current simplistic stereogram shader, since it doesn't yet implmenent features such as hidden-surface removal (where a part of the model should only be visible by one eye) which in turn leads to additional "echoes" in the image.

I've always been fascinated by stereoscopic images and their ability to convey depth from just two dimensions, and I was interested to explore their effectiveness in implementing a Surface application for 3D visualisation.

As a multiuser and multidiretional platform, Microsoft Surface is ideal for viewing 2D content. Since stereoscropic images can viewed¹ from angles orthogonal to an axis designed to align with the plane of the viewer's eyes, they enable depth perception from two opposing viewing positions on either side of a Surface device.

The first step was to generate a stereoscopic image from a depth-map by implementing a pixel shader to render an autostereogram. My initial algorithm was very basic, and produced images as per the example shown below in Figure 1.

Figure 1. Initial autostereogram rendering from depth map.

The initial shader takes the following parameters:

• Depth texture
• Tile texture
• Size of tile (number of vertical strips)
• Depth factor

¹In order to perceive a 3D image the viewer must decouple convergence and focusing of their eyes. In order to aid convergence, a disc and ring have been placed at the correct separation. Looking "through" the image results in four shapes (a disc and ring from each eye). The eyes are correctly converged when the two centre shapes "overlap" to give a disc within a ring. At this point the eyes must be refocussed without changing their convergence. Bright light can help, since a contracted iris gives a decreased depth of field.