My journey developing an app for Autistic kids in India

It was December 2017, the penultimate semester of my undergrad was over and I was placed in a great company hrough the campus placement. However the feeling of not having contributed much to the society was nagging me somewhere. While I was browsing through my IITD mailbox, I jumped to take part in the first opportunity I found. Several projects were being taken up by The National Trust of India and IITD in a joint venture. I signed up for it.

It turned out among the many projects to spread awareness about autism and promote inclusiveness, one of them was to develop an Android app. This app would help children children having autism communicate better using a flashcard like interface. I, along with a team of 5 others were selected for this project. It turned out there were already some apps existing for this very same problem, but they were not “Indianized” enough. The requirement was to have an app that would have the local variations of the common words (eg. “biscuit” instead of “cookeies”) and have its UI primarily in Hindi. We also wanted the app to be free of charge so that families from any financial backgroud could use it.

Planning the app

Our first step was to understand the problem and the requirements of the app. Naturally we decided to meet the teachers of a special school, Action For Autism. We went there for a day, observed the kids and how they interacted with the teachers. We then also used the current similar apps in the market and took feedback from the teachers and parents. As a result we had several points, features and a good idea of how we should proceed with the development.

We realized we needed to solve the following:-

  • Have an app which helps kids communicate by tapping on word tiles in app
  • Keep the words in India context
  • Support the Hindi language for UI
  • Be customizable

We brainstormed that these could be solved by:-

  • Having a decluttered and very simple UI
  • Have a default Text to Speech (TTS) for each tile to speak the word out.
  • Do a comprehensive study of commonly spoken words and actions by Indian kids and include them in the app.
  • Allow kid’s caretaker to add new word tiles
  • Allow chaning of picture on the tile
  • Allow recording of voice for a tile

I being the only one with a Computer Science background in the team decided to take up the responsibility for overseeing the app development.

The App

The app can be broken down into the following main components:-

  • The user interface
  • Text to Speech
  • Data storage
  • UI customizations
  • Content customizations
  • Language

The user interface

The other members in my team were primarily responsible for brainstorming and coming up with a simple, intuitive UI. We came up with some sketches and Parth Dixit also coded up a sample screen in Android Studio

Home page

First sketch of the home page First sketch of the home page

The word tiles and the folder tiles are separated. The word tiles are places in the upper half of the screen. Pressing them plays the word. Pressing a folder tile, on the lower half of the screen, opens the folder and displays its word tiles on the upper half. This is just like a normal directory structure.

Settings menu, with all the options. (The child mode option is missing here)

Settings menu Settings menu

The dialog box allowing user to create a new tile

Create a new tile Create a new tile

Settings page

Settings page Settings page

This was the UI idea we started with and built over.

Text to speech

The next important thing to get right was the Text-To-Speech (TTS). We wanted this to have an Indian accent so that Indian kids could comprehend it easily. It turned out the out of the box TTS system by Android satisfied our needs for this.

myTTS = new TextToSpeech(ctx, this);
myTTS.setSpeechRate(Utility.getSpeechRate(ctx));
myTTS.setLanguage(new Locale("en", "IN"));
myTTS.speak(word, TextToSpeech.QUEUE_FLUSH, null, "sentSpeak");

This was enough to speak out english words with an Indian accent.

Apart from the default TTS, we also wanted to allow the user to record custom voice for the tiles. But now consider this: The user creates a sentence by pressing some tiles with default TTS and some with a custom recording. We need to speak this sentence out too. For this I wrote a SpeakSentence wrapper. This would be passed a stack of the word tiles and would alternate between custom and default TTS as needed.

private void speakNextWords() {
	// Check how many words have been spoken
    if(curWordIndex >= words.size()) {
        sel.sentenceSpoken();
    }
    else if(words.get(curWordIndex).getSoundPath().equals("")) {
    	// If no custom voice exists
        String curString = "";

        // Generate the sentence using all words ahead
        // that require TTS
        while (curWordIndex < words.size() && words.get(curWordIndex).getSoundPath().equals("")) {
            curString += (" " + words.get(curWordIndex).getTitle());
            curWordIndex++;
        }

        // Speak the words
        speakWords(curString);
    } else {
    	// We have a custom voice for the word
        while (curWordIndex < words.size() && !words.get(curWordIndex).getSoundPath().equals("")) {
        	// Play the voice using media player
            MediaPlayer mp = words.get(curWordIndex).playSound(ctx);
            mp.setOnCompletionListener(new MediaPlayer.OnCompletionListener() {
                @Override
                public void onCompletion(MediaPlayer mp) {
                    Log.d(TAG, "Spoken this stuff");
                    speakNextWords();
                }
            });
            curWordIndex++;
            break;
        }
    }
}

Data storage

The data is stored locally using the SQLite DB provided by Android. I created 2 classes, DataProvider and DBHelper for the same.

DataProvider initializes the database when the app is first launched. It reads the words from .txt files. A file for each folder. Each file having comma separated words. The other parts of code interact with DataProvider and it acts as a wrapper over DBHelper.

DBHelper interacts with the SQLite DB and runs the SQL commands as required. Some functions are:-

static void createDatabase(Context ctx) {
    SQLiteDatabase database = ctx.openOrCreateDatabase(DB_NAME, Context.MODE_PRIVATE, null);
    database.execSQL("CREATE TABLE IF NOT EXISTS FolderTiles(id INTEGER PRIMARY KEY, " +
            "title TEXT, parent_folder INTEGER, image_path TEXT, sound_path TEXT);");
    database.execSQL("CREATE TABLE IF NOT EXISTS WordTiles(id INTEGER PRIMARY KEY, " +
            "title TEXT, parent_folder INTEGER, image_path TEXT, sound_path TEXT);");
    database.close();
}

static List<WordTile> getWordsInside(int folderID, Context ctx) {
    List<WordTile> list = new ArrayList<>();
    SQLiteDatabase database = ctx.openOrCreateDatabase(DB_NAME, Context.MODE_PRIVATE, null);
    // Get all words within the given folder
    Cursor cursor = database.rawQuery("SELECT * FROM " + WORD_TABLE_NAME +
            " WHERE parent_folder = " + folderID + ";", null);
    cursor.moveToFirst();

    // Create WordTile objects using all the retrieved rows
    while(!cursor.isAfterLast()) {
        WordTile wordTile = new WordTile(cursor.getString(1), cursor.getString(3),
                cursor.getString(4), cursor.getInt(0));
        list.add(wordTile);
        cursor.moveToNext();
    }
    cursor.close();
    database.close();
    return list;
}

UI customizations

The settings page of the app allows user to make some basic customizations. We allow 2 types of customizations:

  • Font size: The user can select among 3 font size settings, making the text small or bigger as per convenience.
  • Speech rate: This is the speed at which TTS speaks out the words. This again comes with 3 values among which the user can select.

The value selected for these customizations is stored in SharedPreferences. The interaction with SharedPreferences is done by the Utility class. While rendering the screen or speaking out TTS, the Utility class is again consulted for these values.

public static float getFontSize(Context ctx) {
    SharedPreferences prefs = ctx.getSharedPreferences(Utility.PREFS_NAME, 0);
    if(!prefs.contains("font_size")) {
        SharedPreferences.Editor editor = prefs.edit();
        editor.putInt("font_size", FONT_MED); editor.commit();
        return FONT_MED;
    } else {
        return prefs.getInt("font_size", FONT_MED);
    }
}

public static void setFontSize(float size, Context ctx) {
    SharedPreferences prefs = ctx.getSharedPreferences(Utility.PREFS_NAME, 0);
    SharedPreferences.Editor editor = prefs.edit();
    editor.putInt("font_size", (int) size); editor.commit();
}

Content customizations

An important feature of the app would be to allow the users to customize the default words and folders present in the app, and also add more of these. On the user interface aspect, we decided that when the child mode is disabled, long pressing any tile would present a pop-up menu. This would allow the user to change the tile text, image, sound or delete it completely.

Tile edit box

Tile edit box Tile edit box

Edit user flow Edit user flow

To allow these customizations, we needed to modify the data stored in the SQLite database as discussed before.

static void changeImageOfTile(String imagePath, Tile tile, Context ctx) {
    if(tile instanceof WordTile) {
        SQLiteDatabase database = ctx.openOrCreateDatabase(DB_NAME, Context.MODE_PRIVATE, null);
        database.execSQL("UPDATE " + WORD_TABLE_NAME + " SET image_path = ? WHERE id = ?;",
                new Object[] { imagePath, tile.getID() });
    } else if(tile instanceof FolderTile) {
        SQLiteDatabase database = ctx.openOrCreateDatabase(DB_NAME, Context.MODE_PRIVATE, null);
        database.execSQL("UPDATE " + FOLDER_TABLE_NAME + " SET image_path = ? WHERE id = ?;",
                new Object[] { imagePath, tile.getID() });
    }
}

static void deleteTile(int id, Context ctx) {
    SQLiteDatabase database = ctx.openOrCreateDatabase(DB_NAME, Context.MODE_PRIVATE, null);
    database.execSQL("DELETE FROM " + WORD_TABLE_NAME + " WHERE id = " + id + ";");
}

Language

It was important that the app have text in hindi. We initially ad just the words and folders in hindi, but upon taking feedback from teachers, we were told they really need an app that was completely in Hindi. This was a mojor requirement and no app in the Play Store that satisfied it. Hence we were determined to come up with a solution for this too.

Going by the recommended route, I created a new values folder with the name values-hi with a strings.xml file in it for hindi translations. Then using the Translations Editor in the Android Studio, I was able to add the translations for all the required sentence in the app UI.

The option to select a language was added in the settings menu and this too was stored in SharedPreferences in with the Utility class being a wrapper over SharedPreferences.

Whenever the app is launched we set the locale using

public void setLocale() {
	// Get the language set by the user
    String lang = Utility.getLanguage(this);

    // Set the locale
    Locale myLocale = new Locale(lang);
    Resources res = getResources();
    DisplayMetrics dm = res.getDisplayMetrics();
    Configuration conf = res.getConfiguration();
    conf.locale = myLocale;
    res.updateConfiguration(conf, dm);
}

The strings are then accessed by

ctx.getResources().getString(R.string.enter_tile_name)

or

<item android:id="@+id/menu_add_folder"
    android:title="@string/menu_new_folder"/>

Concluding

This was a great project. We created an app which will be used by many kids who need such a solution. We learnt the importance of feedback while developing a product and having a very clean and simple UI.

I’m really thankful to my team, PVM Rao Sir and The National Trust for this opportunity.


© 2018. All rights reserved.

Powered by Hydejack v7.5.0