J2i.Net

Nothing at all and Everything in general.

Passing thoughts, video effects on Windows Phone

Some years ago I saw the movie "A Scanner Darkly." There isn't much to talk about as much as the plot goes, but the visuals of the movie were unique. The movie was done with real actors but it the look of everything was as though it were drawn like a cartoon.

I thought about making an application that would allow some one to produce a similar effect in real time (or close to it) using a phone's camera. I thought I would be able to implement it with the K-means algorithm operating within color space (I will do another post on the details of this). Before diving into this task I needed to make sure that the phone was capable of doing this. I started by taking a look at Windows Phone and these were the main things that I needed to be able to satisfy:

  • Is real time access to the camera available
  • Can I render video frames to the screen at a rate
  • Can the phone provide the computational capability to quickly do the image processing

One of the new capabilities that comes with the Mango update to Windows phone is access to the camera. In addition to getting information from the camera through tasks (which was available with the initial release of Windows Phone) Microsoft has granted developers the ability to paint a surface with a live feed from the camera, capture a video from the camera, capture a frame from the preview buffer, and take a photograph (without user interaction) from the camera. Let's examine how each one of those features does or does not contribute towards my goal and the program design.

Because of the the nature of my goal (to work with video) the Windows Phone Tasks (Camera Capture and Photo Chooser) won't work for my program. They both require user interaction for each frame captured. That's no way to work with video.

What about taking pictures automatically? This doesn't quire work either. Picture taking is slow. In general you'll find that the CCDs used in many digital devices are now able to capture and transmit the information from a full resolution photograph as quickly as they do when sending lower resolution video.

The ability to display the video buffer on screen looks promising. With it you can display what ever the camera sees. However this capability is only for displaying the camera's "vision" on the screen and rendering over it (such as in augmented reality).

This leaves two methods left: using the preview buffer and using the phone's video capturing abilities. Using the phone for video capture gives the highest framerate but it ceases to be real time. I'd be fine with that. That would just mean that some one would need to film a video and then it would play back with the video affect applied. But that would also require that I decode the resulting MP4 video myself (there's no video codec available to do this). So the preview buffer seemed like the best option. So I did a quick test to see how many frames I could capture per second (before performing any processing).

 

public MainPage()
{
    InitializeComponent();
    _camera = new PhotoCamera();
    _camera.Initialized += new EventHandler<CameraOperationCompletedEventArgs>(_camera_Initialized);            
    videoBrush.SetSource(_camera);                 
}

void _camera_Initialized(object sender, CameraOperationCompletedEventArgs e)
{
    var x = _camera.PreviewResolution;
    int pixelCount = (int) (x.Width*x.Height);
    buffer = new int[pixelCount];
    _camera.PreviewResolution
    Dispatcher.BeginInvoke(() => { });

    Thread ts = new Thread(new ThreadStart(GrabFrames));
    ts.Start();
}

void GrabFrames()
{
    _camera.GetPreviewBufferArgb32(buffer);
    var startDate = DateTime.Now;
    for(int i=0;i<100;++i)
    {
        _camera.GetPreviewBufferArgb32(buffer);
    }
    var endTime = DateTime.Now;
    var delta = endTime.Subtract(startDate);
}

The results I got back on a Mango Beta HD7 worked out to 10 frames per second. Not quite real time video. So it looks like my best option is to go with the MP4 video recorder. I'll have to figure out how to read frames from an MP4 file.

I'm glad I was able to figure that out before writing a substantial amount of code or doing a substantial amount of design.

Displaying an Image from Isolated Storage

I was writing some code for someone and for the scenario I needed to display images that were stored in isolated storage. Unfortunately at present the Image element in Silverlight doesn't support displaying images from Isolated storage. You must load the image yourself and then give the loaded image to an Image element. At first glance this is annoying because I wanted to just give the image a URI to something stored in solated storage and let the element do it's magic. After a few minutes of thinking I realized I can still do this provided I create a converter to do some of the needed work.  I wanted to ensure the converter was both compatible with the URIs that point to isolated storage and those that refer to locations on the internet. 

public class FilePathToImageConverter: IValueConverter
{
    private static IsolatedStorageFile ISF = IsolatedStorageFile.GetUserStoreForApplication();

    public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
    {
        string path = value as string;
        if (String.IsNullOrEmpty(path))
            return null;
        if((path.Length>9)&&(path.ToLower().Substring(0,9).Equals("isostore:")))
        {
            using (var sourceFile = ISF.OpenFile(path.Substring(9), FileMode.Open, FileAccess.Read))
            {
                BitmapImage bm = new BitmapImage();
                bm.SetSource(sourceFile);
                return bm;
            }
        }
        else
        {
            BitmapImage bm = new BitmapImage(new Uri(path));
            return bm;
        }
    }

    public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
    {
        throw new NotImplementedException();
    }
}

To use it bind a string value to the Source property of the Image element. The file path must be prefixed with "isostore:" for the converter to extract the image from isoated storage. if the "isostore:" prefix is not present the converter will attempt to load the image using a regular Uri.

bada 2.0 Released

bada is Samsun's operating system for lower priced phones (note: the lowercase 'b' is not a type, it is always spelled in lower case). I had interest in an earlier version of bada but there was one thing that kept me away; the SDK did not have support for 64-bit computers. You could run it on 64-bit computers but it would creash frequently and behave in odd ways. Version 2.0 of bada was recently release and one of the new features the SDK has is that it will work on 64-bit computers. I am downloading the SDK now and hope to get started with it soon. 

You can find more information and the SDK download from bada.com.

StartUp Camp

This weekend I went to an event known as "StartUp." Apparently these events are not uncommon, but it was new to me and was an interesting experience. In a Start-up weekend people present their ideas for an application to developers, artists, and people of other skills. After the presentation of the initial ideas people organize themselves into teams and try to make the application a reality.

The event ran from Friday to Sunday. Of those three days I was only present on Sunday to help some people out and ended up being dedicated to a single team helping them out with their Windows Phone coding. At the end of the event every one does their presentations on what they came up with to a panel of judges. There's also people there that are ready to fund projects. I wish I had known of events like this before. These are perfect opportunities for people that have application ideas but can't make the application by themselves. 

Slides from What's New in Windows Phone Mango

Last month I did a presentation at the Macon .Net user group on what's new in Windows Phone Mango. I also presented this at the Atlanta Windows Phone developer's group. The slides were uploaded to Skydrive a few days ago but I never did post the link. So here it is. You can download the slides and the example code from here.

Windows Phone NTP Client

Download Code (53 KB)

I've got plans for an application in which I need to know with certainty the current time. So I don't want to trust the time as reported by the user's device. Instead I need to retrieve the time from a server. I made a NTP (Network Time Protocol) client for retrieving this information. This is something that couldn't be done on Windows Phone 7 prior to Mango. But with the fall update (Mango) access to sockets is granted.

The use of the client is pretty simple. At minimum one can create the client with the default constructor, subscribe to the ReceivedTime event, and call the RequestTime method to initiate a request. The ReceivedTime event may be called on a thread other than the UI thread so remember to use a dispatcher when making UI updates.

NTP Client

This is an example of a client using the code to display both the system time and the network time.

public partial class MainPage : PhoneApplicationPage
{
    private NtpClient _ntpClient;
    public MainPage()
    {
        InitializeComponent();
        _ntpClient = new NtpClient();
        _ntpClient.TimeReceived += new EventHandler<NtpClient.TimeReceivedEventArgs>(_ntpClient_TimeReceived);
    }

    void _ntpClient_TimeReceived(object sender, NtpClient.TimeReceivedEventArgs e)
    {
        this.Dispatcher.BeginInvoke(() =>
                                        {
                                            txtCurrentTime.Text = e.CurrentTime.ToLongTimeString();
                                            txtSystemTime.Text = DateTime.Now.ToUniversalTime().ToLongTimeString();
                                        });
    }

    private void PhoneApplicationPage_Loaded(object sender, RoutedEventArgs e)
    {
            
    }

    private void UpdateTimeButton_Click(object sender, RoutedEventArgs e)
    {
        _ntpClient.RequestTime();
    }
}

XNA Animated Sprite code uploaded to CodeProject.com

I've uploaded some code I was working on to animate sprites in XNA.

Animating a sprite isn't difficult, but I wanted some way to animate them but reduce the coupling between code and the animation. The Content Pipeline is perfect for this. So I created a component that will handle the animation scenarios that I need along with a content extension so that I could load these animations as content. Right now the animation information is in an XML file. This is a stepping point towards having a graphical tool for handling this.

You can read about the code here or see a brief description of it in the video below

Site Mirror

It seems that my hosting provider has gone through some changes for the worst; once a week for the past three weeks this site has gone offline because of some failure or data loss at my provider's location. Because of the decrease in reliability I'll be looking for a new provider. In the mean time I've started mirroring my content at http://j2inet.wordpress.com/. Any new content I write will also be published there (I may just use WordPress as the primary host for this site, still undecided). If this site ever goes down remember you can see the content there too.

Mango Beta 2 Available for Phones Today!

The Beta 2 Mango Windows Phone Tools are available to developers today! Included with the beta is the ability for developers registered with the AppHub to flash their retail devices.

I know there are some non-developers out there that want to also flash their phones and they may wonder how they get get their phones reflashed with the Mango beta. For the time being they cannot. There is an inherent risk in reflashing the phone; you could end up with a bricked phone if something goes bad. If this happens Microsoft has budgeted to take care of repairing up to one phone per developer. But Microsoft doesn't see this risk as being appropriate for user audiences. [Some] developers on the other hand are willing to risk their device's life and limb to have early access to something new. If you brick your device today Microsoft won't be prepared to act on it for another couple of weeks. That's not the best case scenario. But the alternative was to wait another couple of weeks before releasing the Mango tools. If you don't feel safe walking the tight rope without a safety net then don't re-flash your device yet.

According to the Windows Phone Developer site if you are a registered developer you will receive an e-mail inviting you to participate in early access to Mango.

Changing the Pitch of a Sound

I got a tweet earlier today from some one asking me how to change the pitch of a wave file. The person asking was aware that SoundEffectInstance has a setting to alter pitch but it wasn't sufficient for his needs. He needed to be able to save the modified WAV to a file. It's something that is easy to do. So I made a quick example

Video Example

I used a technique that comes close to matching linear interpolation. It get's the job done but isn't the best technique because of the opportunity for certain types of distortion to introduced. Methods with less distortion are available at the cost of potentially more CPU cycles. For the example I made no matter what the original sample rate was I am playing back at 44KHz and adjusting my interpolation accordingly so that no unintentional changes in pitch are introduced.

To do the work I've created a class named AdjustedSoundEffect. It has a Play() method that takes as it's argument the factor by which the pitch should be adjusted where 1 plays the sound at the original pitch, 2 plays it at twice its pitch, and 0.5 plays it at half its pitch.

If you are interested the code I used is below.

using System;
using System.IO;
using System.Net;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Documents;
using System.Windows.Ink;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Animation;
using System.Windows.Shapes;
using Microsoft.Xna.Framework.Audio;

namespace J2i.Net.VoiceRecorder.Utility
{
    public class AdjustedSoundEffect
    {
        //I will always playback at 44KHz regardless of the original sample rate. 
        //I'm making appropriate adjustments to prevent this from resulting in the
        //pitch being shifted. 
        private const int PlaybackSampleRate = 16000;
        private const int BufferSize = PlaybackSampleRate*2;

        private int _channelCount = 1;
        private int _sampleRate;
        private int _bytesPerSample = 16;
        private int _byteCount = 0;
        private float _baseStepRate = 1;
        private float _adjustedStepRate;
        private float _index = 0;
        private int playbackBufferIndex = 0;
        private int _sampleStep = 2;

        private bool _timeToStop = false;

        private byte[][] _playbackBuffers;

        public bool IsPlaying { get; set;  }

        public object SyncRoot = new object();


        private DynamicSoundEffectInstance _dse;

        public static AdjustedSoundEffect FromStream(Stream source)
        {
            var retVal = new AdjustedSoundEffect(source);
            return retVal;
        }

        public AdjustedSoundEffect()
        {
            _playbackBuffers = new byte[3][];
            for (var i = 0; i < _playbackBuffers.Length;++i )
            {
                _playbackBuffers[i] = new byte[BufferSize];
            }
                _dse = new DynamicSoundEffectInstance(PlaybackSampleRate, AudioChannels.Stereo);
            _dse.BufferNeeded += new EventHandler<EventArgs>(_dse_BufferNeeded);
        }

        void SubmitNextBuffer()
        {
            if(_timeToStop)
            {
                Stop();
            }
            lock (SyncRoot)
            {
                byte[] nextBuffer = _playbackBuffers[playbackBufferIndex];
                playbackBufferIndex = (playbackBufferIndex + 1)%_playbackBuffers.Length;
                int i_step = 0;
                int i = 0;

                int endOfBufferMargin = 2*_channelCount;
                for (;
                    i < (nextBuffer.Length / 4) && (_index < (_sourceBuffer.Length - endOfBufferMargin));
                    ++i, i_step += 4)
                {

                    int k = _sampleStep*(int) _index;
                    if (k > _sourceBuffer.Length - endOfBufferMargin)
                        k = _sourceBuffer.Length -endOfBufferMargin ;
                    nextBuffer[i_step + 0] = _sourceBuffer[k + 0];
                    nextBuffer[i_step + 1] = _sourceBuffer[k + 1];
                    if (_channelCount == 2)
                    {
                        nextBuffer[i_step + 2] = _sourceBuffer[k + 2];
                        nextBuffer[i_step + 3] = _sourceBuffer[k + 3];
                    }
                    else
                    {
                        nextBuffer[i_step + 2] = _sourceBuffer[k + 0];
                        nextBuffer[i_step + 3] = _sourceBuffer[k + 1];

                    }
                    _index += _adjustedStepRate;
                }

                if ((_index >= _sourceBuffer.Length - endOfBufferMargin))
                    _timeToStop = true;
                for (; i < (nextBuffer.Length/4); ++i, i_step += 4)
                {
                    nextBuffer[i_step + 0] = 0;
                    nextBuffer[i_step + 1] = 0;
                    if (_channelCount == 2)
                    {
                        nextBuffer[i_step + 2] = 0;
                        nextBuffer[i_step + 3] = 0;
                    }
                }
                _dse.SubmitBuffer(nextBuffer);
            }
        }

        void _dse_BufferNeeded(object sender, EventArgs e)
        {
            SubmitNextBuffer();
        }

        private byte[] _sourceBuffer;
        

        public AdjustedSoundEffect(Stream source): this()
        {
            byte[] header = new byte[44];
            source.Read(header, 0, 44);

            // I'm assuming you passed a proper wave file so I won't bother 
            // verifying  that  the  header  is properly formatted and will 
            // accept it on faith :-)

            _channelCount = header[22] + (header[23] << 8);
            _sampleRate = header[24] | (header[25] << 8) | (header[26] << 16) | (header[27] << 24);
            _bytesPerSample = header[34]/8;
            _byteCount = header[40] | (header[41] << 8) | (header[42] << 16) | (header[43] << 24);
            _sampleStep = _bytesPerSample*_channelCount;
            _sourceBuffer = new byte[_byteCount];
            source.Read(_sourceBuffer, 0, _sourceBuffer.Length);


            _baseStepRate = ((float)_sampleRate) / PlaybackSampleRate;
        }

        /// <summary>
        /// 
        /// </summary>
        /// <param name="pitchFactor">Factor by which pitch will be adjusted. 2 doubles the frequency,
        /// // 1 is normal speed, 0.5 halfs the frequency</param>
        public void Play(float pitchFactor)
        {
            _timeToStop = false;

            _index = 0;
            lock (SyncRoot)
            {
                _adjustedStepRate = _baseStepRate * pitchFactor;
                _index = 0;
                playbackBufferIndex = 0;
            }
            if(!IsPlaying)
            {
                SubmitNextBuffer();
                SubmitNextBuffer();
                SubmitNextBuffer();
                _dse.Play();
                IsPlaying = true;
            }
        }

        public void Stop()
        {
            if(IsPlaying)
            {
                _dse.Stop();
            }
        }
    }
}