Camera Control

By
Dave
2012
Jan
31
17:58
Posted in

Currently the app supports a number of different types of manipulation, depending on the camera mode. These are demonstrated in the following short video and summarised below.

Video 1. Touch Manipulation used for camera control, recorded on a Microsoft Surface 2.0.

Free Camera

  • Pitch and Yaw by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background

Orbit Camera

  • Orbit body by moving one or more touchpoints on the background
  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Geosync Camera

  • Roll by rotating two or more touchpoints on the background
  • Adjust distance to body by pinch-zooming two or more touchpoints on the background

Tracking Camera

  • Roll by rotating two or more touchpoints on the background

As shown in the video, when time is running, an orbit camera travels with the body, maintaining constant orientation so that the background stars remain fixed. The geosync camera always points at a specific coordinate on the body, and maintains a constant north bearing.

Smoothing

Touch resolutions oftern correspond to approximately the screen resolution in pixels, hence smoothing is necessary to avoid "jumps" in orientation or position. Also of importance is the use of momentum and inertia to provide a more "natural" touch experience.

I initially used a simple spring algorithm to add smoothing, momentum and inertia, and tracked manipulation speed to continue inertia when manipulation ended. This worked well at high framerates, but when using vertical sync (i.e. 60fps) the experience degraded.

Switching to a simple linear interpolation (LERP) of position and spherical linear interpolation (SLERP) for orientation bahaves well at lower frame-rates, and also gives the impression of momentum and inertia. I no longer track manipulation speed, nor continue inertia when the interpolation is complete.

Orientation

By
Dave
2011
May
17
20:48
Posted in

Since I wanted to support running the application on a Microsoft Surface device, it was important to cater for multi-directional interaction. User interface (UI) components can eiher be directional or not. For example, the camera view is non-directional in the sense that it is valid for all users regardless of direction (it is perfectly acceptable to be "upside down" when rolling through 180° in a 3D environment). Text labels, on the other hand, are directional and most beneficial when directly facing a user.

The following screenshots illustrate how the user interface supports multiple directions and multiple users.

Orientation

Figure 1. User interface shown orientated for single user.

Orientation

Figure 2. User interface components are shown with varying orientation to support multiple users.

Figure 2 shows a menu and reticle for each of two users on opposite sides. While there can be many instances of menus and reticles at any one time, a given instance is typically used by one user at a time. It is therefore possible to orient them to the relevant user, either by using the orientation of a Surface tag, or by using the orienation reported by the API for a touch event.

For items such as background labels which are shared between multiple users, it is necessary to pick a consistent orientation for all instances. This "system orientation" can either be determined in code (e.g. by examining the orientation of other directional UI components) or by a user via a menu setting. In Figure 2 the orienation has been chosen to face one of the two users.

While the system orientation is an analogue value (i.e. the background labels, for example, can face any consistent direction), it makes sense to axis-align the orientation of items such as the clock to a side of the screen.

Surface 3D Demo Part 3

By
Dave
2011
Mar
18
20:32
Posted in

In Part 1 and Part 2 I described an interactive autostereogram viewer. In this post, I thought I start by including a link to a video.

The video shows how multitouch gestures on Microsoft Surface can be used to interact with the models. A depth-map shader then processes the model, and a second-pass converts the depth-map to an autostereogram. As described previously, to avoid distracting the eye when animating a tile-based autostereogram, a randon-dot pattern is used which is regenerated each frame. Unfortunately the video doesn't have the required resolution to see much detail when animating the autostereogram, but the 3D-effect seen when the animation stops is maintained when viewing the application.

Here are some further screenshots:

SIRDS

Figure 1. Depth map

SIRDS

Figure 2. SIRDS for animation

SIRDS

Figure 3. Texture for static image

Multitouch Game Loop Input Processing

By
Dave
2011
Jan
21
19:37
Posted in

There are multiple potential sources of input for an XNA game component, such as keyboard, gamepad, mouse, Surface multitouch, .NET 4 Touch, etc. Dealing with multpile approaches to gathering input in the Update loop of each component can become complex, and adding support for a new input type further adds to this complexity.

I divided my input handling into two different categories:

  1. Controller-based input such as keyboard and gamepad.
  2. Manipulation-based input such as mouse, Surface multitouch, and .NET 4 touch.

For the latter, I wanted to abstract input handling such that a given game component only had to deal with a custom Manipulator type. In this way, I would be free to change or add input source(s) without affecting the implementation in a game component.

Clearly, different input sources provide different types and degrees of information. Surface multitouch Contacts for example, provide information on size, orientation, type etc, whereas a mouse only provides position. In many cases only position information is necessary, however additional properties can easily be added to the Manipulator type and supported by a game component if available. In this case I decided to sub-class my Manipulator type to the following:

  1. Mouse and finger-touch
  2. Surface Tag contacts
  3. Surface blob contacts

In order to deal with multitouch manipulations, I could process input using my previosuly-discussed Manipulation classes and processor.

The following video demonstrates the use of multiple input sources:

Video 1. Input processing from multiple sources.

Since a mouse can only deliver single-touch input, when demonstrating mouse input in Video 1 I switch the pivot type to "Tracked" in order to be able to demonstrate rotation and scaling, as shown in Figure 1.

Figure 1. Input processing from multiple sources.

Of course, it is unlikely that a scenario mixing both Mouse and Surface input would ever be used in practice, however it serves to illustrate how a game component can handle input in a consistent way, without being aware of the input source. For example, the buttons on the left of the screen are "pressed" using either Surface v1 Contact or a mouse-click, however the buttons only tracking Manipulator state.

A useful application of this approach is the ability to write XNA applications which work with both Surface v1 multitouch input and .NET 4 multitouch (and mouse for single-touch if multi-touch hardware is not available) without any code changes, i.e. multi-platform targetting.

Multitouch Game Loop Inertia Processing

By
Dave
2010
Sep
21
18:11
Posted in

In Part 1 I described a lightweight class to process general multitouch manipulations using a game-loop model in XNA.

A key aspect in bringing realism to a multitouch experience is the use of momentum and inertia. The Microsoft.Surface.Core.Manipulations.Affine2DInertiaProcessor provides this support in an event-driven model. In order to continue the approach of using a game-loop model, I needed to add linear and angular velocities, and expansion rates to my manipulation processor.

Changes in manipulator position occur relatively rarely in relation to the game loop frequency, hence calculating rates of change requires averaging these changes over time. The choice of duration and quantity of these time periods is somewhat subjective, and I settled on 5 samples averaged over a maximum of 200ms. Clearly this leads to a small latency between measured and actual velocities, however the smoothing effect is beneficial.

In order to visualise these rates, both for debbugging and to settle on suitable sampling settings, I added visual indicators of direction and magnitude, as shown below in Figure 1.

Multitouch manipulation

Figure 1. Visualising linear and angular velocities and expansion rates during multitouch manipulation.

With the addition of linear and angular velocities and expansion rates, I could now add a simple inertial processing component which uses these properties as input parameters. This inertial processor uses a simple deceleration algorithm, with configurable rates for translation, rotation, and expansion. The solution is demonstrated in the following video.

Video 1. Multitouch game loop inertia processing.

Note that for gestures such as a flick, where velcity increases rapidly, a time-weighted average may be more suitable for calculating the inertia processor's input parameters, and I'll investigate this at a later date.

Multitouch Game Loop Manipulation Processing

By
Dave
2010
Aug
12
23:05
Posted in

Multitouch Surface support for XNA-based applications is provided via the Microsoft.Surface.Core.Manipulations namespace in an event-driven model. I was interested in investigating general multitouch processing within a game-loop model, i.e. not involving events, and thought it would be instructuve to develop my own class to process manipulations in this way.

In a similar manner to the Affine2DManipulationProcessor, I wanted to support arbitrary combinations of translation, rotation, and scale. Translation is trivial, since it is simply a matter of tracking the changes in each manipulation point between calls and taking the average.

Rotation and scale is a little more tricky, since each of these values are relative to a centre-point, or manipulation origin. Rotation is calculated from the average angle between each point and the origin, and scale is calculated from the average distance. Unlike a mouse cursor, manipulation points come and go, so the key is only to track changes to those points which persist between calls, and then update the positions, origin, distance and angles for the next call.

In order to generalise manipulation points from multiple sources such as mouse, gamepad, Surface Contacts and .NET 4.0 Touch, I created my own Manipulation structure to abstract properties such as ID and position. My input processing can then build a collection of these objects from any relevant source(s) and pass them to my manipulation processor as part of the Update call as follows:

myManipulationProcessor.ProcessManipulators(myManipulators);

The cumulative transforms for translation, rotation and scale (according to the supported manipulation types specified) are then immediately available as properties on the manipulation processor.

In order to test the solution, I wrote a small harness to visualise the manipulation points, and their effect on a reference cross-hair. While I also added support for a mouse (right-click to add/remove points, left-drag to move points), multi-touch hardware is required to test scenarios when multiple points are added, moved, or removed simultaneusly. A screenshot is shown in Figure 1.

Multitouch manipulation

Figure 1. Multitouch manipulation with averaged manipulation origin.

One of the complexities of working with multiple manipulation points is deciding how to determine the manipulation origin. One option is to simply take the average position of each point, as shown in Figure 1. Another option is to introduce a speicific "pivot point" as the origin and use this to calculate scale and rotation. This pivot point can either be fixed, or tracking the object being manipulated. The latter two options are shown in Figures 2 and 3.

Multitouch manipulation

Figure 2. Multitouch manipulation with fixed pivot point.

Multitouch manipulation

Figure 3. Multitouch manipulation with pivot point centered on manipulation object.

The solution is demonstrated in the following video. Each approach for determining the manipulation origin is illustrated using translation, rotation, and scaling manipulations, initially in isolation and then in combination.

Video 1. Multitouch game loop manipulation processing.

Surface 3D Demo Part 2

By
Dave
2010
Aug
07
11:03
Posted in

In Part 1 I discussed an simple shader which generated autostereograms from depth maps. The next step was to make the application interactive, so that instead of generating an autostereogram from a static depth-map, an interactive 3D model could be used to generate dynamic autostereograms.

To create a depth-map from a model, I added a pixel shader to return the depth value. A similar example can be found in How To: Create a Depth Texture. In order to obtain a sensible range of depth values, I need to calculate the bounds of the model using bounding spheres. I can then set the near and far planes in my projection matrix to correspond to the closest and furthest points in the model respectively. The depth texture can be extracted by rendering to a render target. It can then be passed as a shader parameter to allow autostereograms to be rendered in real time as the model is manipulated.

One of the main issues with animating an autostereogram is that manipulating the model results in changes across areas of the image which do not correspond to the location of the model itself.1 One way around this distracting side-effect is to use a SIRDS, or Single Image Random Dot Stereogram, and alter the pattern each frame in which the model is manipulated.

Instead of passing a pre-defined pattern texture, I generate a random texture on the CPU as follows:

private Texture2D CreateStaticMap(int resolution)
{
    Random random = new Random();
    Color[] colors = new Color[resolution * resolution];
    for (int x = 0; x < resolution; x++)
        for (int y = 0; y < resolution; y++)
            colors[x + y * resolution] = new Color(new Vector3((float)random.NextDouble()));

    Texture2D texture = new Texture2D(GraphicsDevice, resolution, resolution, 1, TextureUsage.None, SurfaceFormat.Color);
    texture.SetData(colors);
    return texture;
}

The shader then tints the black & white image to give the result shown below in Figure 1.

SIRDS

Figure 1. SIRDS for animation

1This is exacerbated by my current simplistic stereogram shader, since it doesn't yet implmenent features such as hidden-surface removal (where a part of the model should only be visible by one eye) which in turn leads to additional "echoes" in the image.

Surface 3D Demo Part 1

By
Dave
2010
Aug
01
19:19
Posted in

I've always been fascinated by stereoscopic images and their ability to convey depth from just two dimensions, and I was interested to explore their effectiveness in implementing a Surface application for 3D visualisation.

As a multiuser and multidiretional platform, Microsoft Surface is ideal for viewing 2D content. Since stereoscropic images can viewed1 from angles orthogonal to an axis designed to align with the plane of the viewer's eyes, they enable depth perception from two opposing viewing positions on either side of a Surface device.

The first step was to generate a stereoscopic image from a depth-map by implementing a pixel shader to render an autostereogram. My initial algorithm was very basic, and produced images as per the example shown below in Figure 1.

Initial rendering

Figure 1. Initial autostereogram rendering from depth map.

The initial shader takes the following parameters:

  • Depth texture
  • Tile texture
  • Size of tile (number of vertical strips)
  • Depth factor

1In order to perceive a 3D image the viewer must decouple convergence and focusing of their eyes. In order to aid convergence, a disc and ring have been placed at the correct separation. Looking "through" the image results in four shapes (a disc and ring from each eye). The eyes are correctly converged when the two centre shapes "overlap" to give a disc within a ring. At this point the eyes must be refocussed without changing their convergence. Bright light can help, since a contracted iris gives a decreased depth of field.

Surface Physics Download

By
Dave
2010
Feb
13
12:15
Posted in

I've had a lot of requests to share my physics-enabled, WPF layout control for Microsoft Surface. This was demonstrated in the previous series of posts, Surface Physics Demo Part 1, Part 2, and Part 3.

So here it is! The following downloads are available:

  1. Physics Library (binary), .dll (zip'd), 19Kb. The physics library and layout control.
  2. Surface Physics Sample (install), .msi (zip'd), 1,046Kb. The sample application for demonstrating the physics library and layout control.
  3. Readme for Surface Physics Sample, .pdf, 1,626Kb. Readme for the sample application.
  4. Surface Physics Sample (source code), Visual Studio 2008 Project (zip'd), 907Kb. Source code for the the sample application.

Note that the Physics Library itself is currenly only available as a binary.

The Surface Physics Sample demonstrates many of the features supported by this library. However, to get you started with using the library in your own projects, I'll discuss how to enable basic physics for a simple ScatterView sample.

Migrating ScatterView to PhysicsView

This example demonstrates how to take a simple ScatterView sample and migrate it to make use of the physics-enabled layout control. You'll need the Microsft Surface SDK, available from the MSDN site here, and access to a Microsoft Surface or at least the Surface Simulator in the SDK. You'll also need the physics library.

First create a new Surface Application (WPF) project using the Visual Studio template inlcuded in the SDK. In SurfaceWindow1.xaml add a ScatterView to the default Grid control as follows:

<s:ScatterView Name="ScatterView1">
    <s:ScatterView.ItemTemplate>
        <DataTemplate>
            <Border BorderBrush="White" BorderThickness="2">
                <Image Source="{Binding}" Width="128" Height="96" />
            </Border>
        </DataTemplate>
    </s:ScatterView.ItemTemplate>
</s:ScatterView>

In SurfaceWindow1.xaml.cs add an event handler for the Loaded event and add some items to the layout control:

public SurfaceWindow1()
{
     InitializeComponent();

     // Add handlers for Application activation events
     AddActivationHandlers();

     this.Loaded += new RoutedEventHandler(SurfaceWindow1_Loaded);
}

void SurfaceWindow1_Loaded(object sender, RoutedEventArgs e)
{
    ScatterView1.ItemsSource = System.IO.Directory.GetFiles(@"C:\Users\Public\Pictures\Sample Pictures");
}

Run the project and if you're running the Surface Simulator you should see the following:

ScatterView

Figure 1. Simple ScatterView example. Note that the blank image is from a hidden file (desktop.ini) in the Sample Pictures folder.

We'll use this as a baseline to migrate from ScatterView to PhysicsView. First of all, copy the physics library (Physics.SurfaceControls.dll) into the project, and add a reference to it. Then add a namespace declaration at the top of SurfaceWindow1.xaml as follows:

xmlns:p="clr-namespace:Physics.SurfaceControls;assembly=Physics.SurfaceControls"

In SurfaceWindow1.xaml do a find and replace on s:ScatterView with p:PhysicsView and change the name from ScatterView1 to PhysicsView1. The rest of the markup remains unchanged, and the layout control should now look like this:

<p:PhysicsView Name="PhysicsView1">
    <p:PhysicsView.ItemTemplate>
        <DataTemplate>
            <Border BorderBrush="White" BorderThickness="2">
                <Image Source="{Binding}" Width="128" Height="96" />
            </Border>
        </DataTemplate>
    </p:PhysicsView.ItemTemplate>
</p:PhysicsView>

We can now set the ItemsSource on the new layout control. The next thing we need to do is inform the physics library of the physical properties of these items. Before we do this, however, we should set up some walls defining the bounding area. The library doesn't make any assumptions here since this area doesn't have to be rectangular nor aligned with the x,y axes. In this case we'll simply add four walls, inset from the extent of the screen by 16px (remembering that a Surface application runs at 1024 x 768px). Update SurfaceWindow1_Loaded to look like this:

void SurfaceWindow1_Loaded(object sender, RoutedEventArgs e)
{
    //ScatterView1.ItemsSource = System.IO.Directory.GetFiles(@"C:\Users\Public\Pictures\Sample Pictures");

    // add walls
    PhysicsView1.Bounds = GenerateWalls(0.5, new Rect(new Point(16, 16), new Point(1008,752)));

    // set data context
    PhysicsView1.ItemsSource = System.IO.Directory.GetFiles(@"C:\Users\Public\Pictures\Sample Pictures");

    // add behaviours
    AddBehaviours(this.PhysicsView1);
}

The methods for setting the bounds and item properties are as follows:

List<WallBody> GenerateWalls(double restitution, Rect rect)
{
    return new List<WallBody> {
        new WallBody { Normal = new Vector(1, 0), StartPoint = rect.TopLeft,
                EndPoint = rect.BottomLeft, Restitution = restitution }, // left
        new WallBody { Normal = new Vector(0, 1), StartPoint = rect.TopLeft, 
                EndPoint = rect.TopRight, Restitution = restitution }, // top
        new WallBody { Normal = new Vector(-1, 0), StartPoint = rect.TopRight,
                EndPoint = rect.BottomRight, Restitution = restitution },  // right
        new WallBody { Normal = new Vector(0, -1), StartPoint = rect.BottomLeft,
                EndPoint = rect.BottomRight, Restitution = restitution }}; // bottom
}

void AddBehaviours(PhysicsView physicsView)
{
    Random random = new Random();

    // ensure layout
    physicsView.UpdateLayout();

    for (int i = 0; i < physicsView.Items.Count; i++)
    {
        // get item
        PhysicsViewItem item = physicsView.ItemContainerGenerator.ContainerFromIndex(i) as PhysicsViewItem;

        // set properties
        Body body = new RectangularBody
        {
            Width = item.DesiredSize.Width,
            Height = item.DesiredSize.Height,
            Density = 0.01,
            InertiaConstant = 0.5,
            Restitution = 0.5,
            Orientation = random.NextDouble() * Math.PI * 2,
            Location = new Point(random.NextDouble() * physicsView.ActualWidth,
                random.NextDouble() * physicsView.ActualHeight),
        };

        // add item
        PhysicsCanvas.SetChildBody(item, body);
    }
}

That's it! The items will now collide with one another. Run the project and you should now see the following:

PhysicsView

Figure 2. PhysicsView Items now collide with one another and do not overlap.

Regrettably I've had no time to extend this work, so a lot of features remain un-implemented at this time. Examples inlcude multi-touch manipulations on the individual items themselves, a better dampening algorithm etc. However, I hope it may still prove useful in some cases.

The source code for this example can be downloaded here.

Surface Physics Demo Part 3

By
Dave
2009
Feb
19
23:52
Posted in

In Part 1 and Part 2 I focussed on 2D features. This makes a lot of sense for Surface applications, as fundamentally items move in two dimensions, however there are particular scenarios that lend themselves to 3D, one of which I'll describe later in this post.

Video 1. Surface physics demo.

The video is divided into the following sections:

Layout

In some cases it is desirable to arrange the items into preset patterns, for example as part of an "Attract Mode" application, or when interacting with physical objects placed on the Surface. This screen defines some basic patterns and "locks" the items to the nearest position in the pattern from their original location. Selecting an item releases the "lock".

3D Rolling Spheres

Spheres lend themselves to an intuitive motion on a 2D-plane, such as when playing marbles, pool etc. When a texture is added to the sphere, it is important to ensure that the sphere "rolls" correctly when moved. Several examples of textures are shown in the video.

Here are some further screenshots.

Marbles

Figure 1. Marbles

Pool

Figure 2. Pool balls

Page