CustomAI - Platform to create custom Image inference api with modus

Before we get started I know you might be having a hard time believing how I made this app in 24 hours. Don’t worry I haven’t slept or moved from my chair and desktop for more than 10 minutes in these past 24 hours. Now let’s get started with the topic of the day. What is customAI, what it does and why did I choose the above-mentioned tech stack to build this project? Let’s get started!

As a student and working professionals, we have to go through lots of projects, and assignments and don’t even have free time left to enjoy life. For those monthly projects in college half of them are about building some cool AI-powered image classification App, for which we have to spend lots of time. Whether disease prediction or classification, dog breed classification, most of the projects done by a student in their lifetime include one of these apps. Talking about our work, most of time we need to test with data and develop some quick test solutions with them. Now let’s discuss what is the problem here and why I made this app.

What made me think about Making Custom AI

Whenever I think about making a project, I think of an app that involves AI. Recently I thought about making an app that helps to diagnose tuberculosis. In my past experiences also I have built a similar type of project like paddy disease prediction, malaria prediction, etc that involves providing labeled data and running inferences, spending hours to train data, and then just getting disappointed at the last minute by creating some half working API. I am thinking here I am not alone and most of you have done that too.

For building those projects, I have to do intense research, search for datasets, and then train the model, and spend days creating API. Then I thought Ahh! So why not just create a platform that takes your label images and performs a vector similarity search to just return your label from the most similar image? This way I can just create those projects in a few minutes and focus my entire time watching series and playing around.

Jokes aside now let’s talk the real-world problems. Let’s say I run multiple businesses. I own an e-commerce store for clothes. I have to organize those clothes, identify which clothes are under what label, and suggest those clothes with similar labels accordingly. For these I have to spend days or even months to create a API that does this. CustomAI can help you with this problem.

The problem customAI deals with is it helps you create ready-to-use API for your labeled image dataset in just a few minutes or hours, which you can then integrate into your applications. It can be your day-to-day projects, assignments, or solving your crucial business needs.

Initial BrainStorming and browsing

I had a rough idea about what I was going to build. Then I started listing the features and started brainstorming. I had to find technical, time, and economic feasibility. Then I came across hypermode modus and dgraph. Features that helped me decide why I chose them:

I had to provide users the ability to query their own data. I couldn’t just make endpoints for everyone. I have to use GraphQL. Then I came across Modus. Code Functions and get ready to use graphql endpoints. What more could I ask for?
I have to store data and my app will be dealing with vector embeddings. Any good database that works well with Modus? Yes, Dgraph. More than that It provided me ready to use graphql API, which I just had to integrate with my Modus application to play with selected data and endpoints. I began questioning discord members from the hypermode channel and started my journey.

Exploring Modus, Dgraph, and going through discord helps

I started going through Modus and dgraph documentation. Browsed everything and here are some key points that I found.

For modus 🛠️

It is an open-source serverless framework for building powerful AI-powered applications.
You can write code either in webAssembly or Go.
You can invoke AI models just like you invoke functions.
Easily secure your endpoints and add authentications.

For Dgraph 🧠⃤

Powerful Graph Database
Dgraph Cloud provides you ready to use graphql API along with features to explore API and data in their platform.
Allows vector embedding storage and embedding relation operations.
Best for AI-powered applications not limited to being vector databases alone but multipurpose ones.

How customAI works

With customAI you can create a project, upload your labeled images, and get ready to use API to run inferences to find similar images and labels. You can create multiple projects and create different API for your different needs. You can also run test for the dataset you uploaded.

Building backend with hypermode, modus and Dgraph

Here is the general workflow for my project creation journey.

Getting started with hypermode and dgraph

Go to hypermode and sign up for your account. Then click Create Project.

Then connect your github account and follow the steps mentioned. You just have to upload github workflow files and it will automatically build backend deployment for you with you commits.

Graphql schema defining in dgraph database

Here is how I defined my Graphql schema.

 type User {
  id: ID!
  clerkId: String! @search(by: [exact])
  email: String
  projects: [Project] @hasInverse(field: user)
}

type Project {
  id: ID!
  name: String
  description: String
  owner: String @search(by:[exact])
  user: User
  datasets: [Image] @hasInverse(field: project)
  isDeployed:Boolean
  endpoint: String
  token: String
}

type Image {
  id: ID! 
  imageUrl: String!
  label: String @search(by:[exact])
  project: Project
  projectId: String @search(by: [exact])
  token: String @search(by: [exact])
  owner:String @search(by:[exact])
  embedding: [Float!] @embedding @search(by: ["hnsw(metric: euclidean, exponent: 4)"])
}

Problem encountered with type defining in dgraph

Then I started building with modus. You can create modus app by installing modus cli and with modus new command. Basic step is given below.

npm install -g @hypermode/modus-cli

modus new

Now follow the questions asked and you are good to go.

For querying graphql api you have to define the connection in the modus.json file.

  "connections": {
    "flaskapi": {
      "type": "http",
      "endpoint": "https://embeddingextractor.com/extract-embedding"
    },
      "dgraph": {
        "type": "http",
        "endpoint": "https://yourapi/graphql",
        "headers": {
          "Authorization": "Bearer {{API_KEY}}"
        }
      }
  }

I have used two api endpoints. One had AI model to extract embeddings and another was dgraph graphql API.

I had faced lots of type mismatched issues between modus queries and dgraph graphql queries. The major ones were between type String! and String, type ID, and [ID!]. I simply checked my schema definition while writing mutations or queries in modus. Here is how I did that.

Here is my project type from graphql schema

type Project { id: ID! name: String description: String owner: String @search(by:[exact]) user: User datasets: [Image] @hasInverse(field: project) isDeployed:Boolean endpoint: String token: String }

All I had to do was mention same type in the first line of mutation statement.

  const statement = `
    mutation updateProject($id: string!, $owner: String!, $name: String!, $endpoint: String!, $isDeployed: Boolean!, $description: String!, $token: String!) {
        updateProject(input: {filter: {id: {eq: $id}}, set: {name: $name, endpoint: $endpoint, isDeployed: $isDeployed, description: $description, token: $token, owner: $owner}}) {
            numUids
        }
    }
`;

I wrote String! as a variable type for a variable that maps to the same type in my schema. Problem solved!

Problem encounter with matching types of data between modus and dgraph graphql API response

Query was working in dgraph graphql API playground but I wasn’t getting any results back for the same query. I was going nowhere due to those null results while querying graphql endpoints in modus. Then after hours of research, checking types and everything I found the one major issue was response wasn’t properly mapped. Here is how I learned how to do so easily.

Go to dgraph and query the data. Now observe properly the returned data. You will see something like this:

{
data: [{
id: “iowahe”,
name: “roshan”
}]
}

Now you have to define class in your modus function at first with @json. If i want those id and name my class will be like this.

@json
class ProjectData {
  id: string;
  name: string;
}

Then as you see the response is an array so create a new response class to get those arrays.

@json
class ProjectResponse {
  queryProject: ProjectData[];
}

Then create a function to query and get a result. Your graphql.execute function will now have ProjectResponse class so it properly maps and returns value.

export function getProjects(userId: string): ProjectData[] {
  const statement = `
    query queryProject($userId: String) {
        queryProject(filter: {owner: {eq: $userId}}) {
           description
            endpoint
            id
            isDeployed
            name
            owner
            token
        }
    }
`;

  const vars = new graphql.Variables();
  vars.set("userId", userId);

  const response = graphql.execute<ProjectResponse>(
    connection,
    statement,
    vars,
  );

  return response.data!.queryProject;
}

This simple thing took me hours to learn and after that took minutes to build my entire enpoints. Hope you will find this useful.

How I added filtered in dgraph graphql API

I was having hard time on how to filter the embedding search to limit it for only the project created by the specific user. Then I found one easy way to add a filter in dgraph graphql. Here is how I did that.

First, go to schema and add @search(by: [exact]) in the attribute by which you want to filter your queries. This indicates you can search by the exact value.

Now I went to graphql tab to play around with queries. That gave me query to use in my modus application for retrieving data.

The last thing remaining was to create an endpoint to filter through this data in modus. Here is how I did that.

import { graphql } from "@hypermode/modus-sdk-as";
import { embeddingGenerate } from "../helpers/embeddings";
import { JSON } from "json-as";

const connection: string = "dgraph";


@json
class LabelData {
  label: string;
}


@json
class QueryImageByEmbeddingResponse {
  querySimilarImageByEmbedding: LabelData[];
}

export function runInference(
  imageUrl: string,
  projectId: string,
  token: string,
): LabelData[] {
  const embedding = embeddingGenerate(imageUrl);
  const data = embedding.embedding;

  const log = JSON.stringify(data);
  console.log(log);
  const statement = `
  query querySimilarImageByEmbedding($embeddings: [Float!]!, $projectId: String, $token: String) {
      querySimilarImageByEmbedding(by: embedding, topK:1, vector: $embeddings, filter: {projectId: {eq: $projectId}, token: {eq: $token}}) {
        label
      }
  }
`;

  const vars = new graphql.Variables();
  vars.set("embeddings", data);
  vars.set("token", token);
  vars.set("projectId", projectId);

  const response = graphql.execute<QueryImageByEmbeddingResponse>(
    connection,
    statement,
    vars,
  );

  const resdata = response.data!.querySimilarImageByEmbedding;
  const logdata = JSON.stringify(resdata);
  console.log(logdata);

  return resdata;
}

Here you can see I created a class for the label and mapped the response. Like this, I implemented for my every endpoints, minimizing errors and rapid developments.

Building Frontend For customAI

The problem that I faced while building the backend was integrating with API, matching types for data sent and received, and everything. It was such a helpless situation. I have to query through Apollo client, write mutations and queries one by one and aah that is such as nightmare.

But guess what building frontend with nextjs was rapid not because I could create components faster but because I came across modus API explorer for the backend API that I created with modus. Here is an example I will show you.

First, run your modus app with modus dev the command go to the API endpoint with http://localhost:8686/explorer

Here you can see options to choose between graphql and Typescript.

I choose typescript. Then I was surprised it provided me necessary codes to make queries. Lack of time, sleepless nights, and what’s even a hackathon if you don’t use hacks like that? I pasted that in my component and made the request.

And with such ease, I could make requests to API endpoints and get data. I began implementing features rapidly in my application after that.

Usage

Currently, you can create image inferences API with customAI. It utilizes vector embedding and similarity search in dgraph database to return labels of similar items. Here is how you can use CustomAI.

Open customAI website and click Get Started.
Log in with your email account. Then click on Create Project. Fill in your project details.
Click on your project. Now you can upload your images and labels one by one. I have done this to prevent abuse of the demo for now.
Then you can test and upload one of the images you provided. This is because you might not have a good labeled dataset.
Get your API and integrate it into your application.

Future of CustomAI with modus and dgraph 🚀🔜

Dgraph and modus have been core technologies for my application. My future thoughts for customAI are providing the ability for people to create their own AI features to use within the application or outside by creating a dedicated API as it does now. Here are some future improvements and additions that I have thought about adding to my project.

💲 Adding a subscription tier to the platform. Implementing credit system for API calls.
⚡ Adding the ability to create multipurpose text-based API with custom data and prompts utilizing the llms provided by hypermode. For example, uploading college notes to generate important questions from own content provided.
💼 The tailored algorithm as per use case: Adding the Ability to choose the best possible way between algorithms or to train custom train models as per need in order to generate API or feature.
🛠️ Adding custom feature creation ability. Not only creating API, user can create and use the features with text and image data of their own, extended by the powerful AI models integrated through modus. For example: User can create their own best health insurance plans finding feature by providing updated policies from their own countries or regions in CustomAI by creating a new project. The project can be the custom features we want, which will work for specific tasks that we want.

Many more things can be added but I have listed these four which can show you future potential.

Technologies used ⚙️

Here are the major technologies that helped me build this project.

Modus and AssemblyScript: A framework that uses typescript-like language and Go programming language and enables you to create powerful AI-powered and or non-AI multi-functional applications.
HyperMode: Platform that allows you to host your modus backend and provides ready-to-use state-of-the-art AI models that can be easily integrated into backends.
Dgraph database and dgraph cloud: A graph database that has a feature to store vector embeddings and perform operations like similarity search and a platform that provides ready-to-use Graphql API.
Nextjs: Full stack framework to develop scalable efficient and powerful web applications typically used to create the front end of web apps.
Clerk: A simple-to-use authentication management service.
Cloudinary: Easy and secure cloud storage service optimized for image data.

Thank you for reading through my blog. I am happy to share my knowledge, learnings, problems encountered and experiences of solving them with you all. Feel free to share it with your friends and have a good holiday!😁🌸

https://youtu.be/L2b00SRj4uE