Tag Archive for: genai

Transforming Mystery Shopping with AI: How HS Brands is Revolutionizing Customer Experience Measurement

/by

HS Brands, a leader in mystery shopping services, demonstrates how AI technology can create compelling competitive advantages.  The approach uses AI to reshape how HS Brands gathers, analyzes and acts on the data supporting customer experience measurement.

Integrating Large Language Models (LLMs) into enterprise workflows has become a crucial strategy for enterprises to enhance efficiency, consistency and quality. A common LLM framework is Retrieval-Augmented Generation (RAG) which focuses on information retrieval and synthesis. However, LLMs offer a broader spectrum of capabilities that can be embedded into otherwise manual business processes to drive significant impact.This experience shines the light on other opportunities to apply AI to traditional business processes while creating strategic advantages throughout an organization.

Industry Background

As businesses prioritize customer-centric strategies, the demand for mystery shopping services has surged, creating opportunities for firms like HS Brands. Many customer-facing industries rely on mystery shoppers to report on their experience when interacting with the business.

These reports often contain a combination of multiple-choice question responses and detailed narratives. HS Brands use editors to ensure that raw shopper reports are complete and consistent, to provide clients with reliable insights. Editors identify discrepancies between dropdown answers and narrative descriptions, then have the shopper reconcile or elaborate in their own voice. Reviewing a single shopper’s complex report is a meticulous task that can consume hours. Integrating AI using LLMs into this process offers a transformative solution to improve efficiency and accuracy.


Challenge: Highly manual review processes

The manual review of mystery shopping reports requires editors to carefully cross-reference dropdown selections with extensive shopper-written narratives. This labor-intensive task is time-consuming, leading to potential delays in report delivery and increased operational costs. In an industry where timely and accurate feedback is essential for client decision-making, these inefficiencies can result in lost opportunities and diminished client satisfaction.

The transformation is particularly evident in the editing process. “Automating many of the editors’ time-consuming tasks, allows them to refocus their brain power towards identifying survey insights,” explains Tommy Mills, CEO of HS Brands. Automating many of editor’s time-consuming tasks, allows them to refocus their brain power towards identifying survey insights

Approach: Use AI to automate time consuming steps in the workflow

Ironside works with clients to identify opportunities for AI in their business and implement in a low risk process. By consulting with business domain subject matter experts and prioritizing AI use cases by rubrics evaluating suitability and impact, the high value uses of AI specific to a client are uncovered. 

The generative AI solution that emerged from Ironside’s process, leverages LLMs to automate the detection of inconsistencies within mystery shopping reports.  This greatly streamlines the editor workflow. Prompted appropriately, the LLM identifies missing responses and possible contradictions between dropdown responses and narrative content, flagging these sections for follow-up by human review.

The prompt’s instructions to the LLM include: 

  • Report reviewing guidelines
  • Sample editor comments
  • Step by step instructions

This custom engineered prompt helps the LLM understand the context and typical patterns of shopper report inconsistencies.  The LLM and core AI functionality is implemented within an Amazon Q Business web app that can process new reports in seconds, rapidly flagging any discrepancies for further human review.  The output from the Amazon Q Business app gives the editors a flying start to thoroughly understand the number and nature of reconciliations needed for the report.  Editors can adapt the editing guidance to help individual shoppers learn how to deliver more consistent and accurate reports. 

Strategic advantages of AI implementation

There are 3 key operational benefits in adopting generative AI:

  1. Enhanced Data Quality – “The editors can focus on more than highly intelligent language, they can drill in on the details, asking shoppers for more detail, and ultimately giving the client a better product,” notes Mills. This improvement in quality creates a compelling differentiation in the market.
  2. Consistent Evaluation Standards – AI provides standardized analysis across different regions and markets, addressing a critical challenge in maintaining quality across diverse locations. The technology ensures that evaluation criteria are applied uniformly, regardless of geography or editor.
  3. Operational Efficiency – The automation of routine tasks leads to significant time savings. As Mills points out, “AI would save a good amount of time…” and indeed early estimates suggest a 25% reduction in the editing cycle.

Creating strategic value for clients

AI implementation delivers more than operational benefits. Through AI, HS Brands offers insights beyond standard mystery shopping offerings.  One example area is Training and Development.  An AI system can identify patterns in customer service delivery and provide actionable insights. “Mystery shopping data does feed training insights,” Mills emphasizes. “Our recommendations can range from micro issues like ‘Team Member X made this mistake at location 5’ all the way to ‘We’ve noticed in Region 7 that we’ve got a much bigger problem.'”

Expanded mystery shopper offerings powered by AI include:

  • Deeper Analytics –
    • Pattern recognition across locations and regions
    • Trend identification in service delivery
    • Predictive insights for training needs
  • Enhanced Training Support – The AI system can:
    • Identify specific training needs by region
    • Create heat maps of performance issues
    • Generate targeted recommendations for improvement
  • Strategic Decision Support – The system provides different insights based on organizational level:
    • Location managers receive specific operational insights
    • District managers get trend analysis
    • Regional managers access strategic patterns
    • Corporate leadership obtains system-wide insights

“Clients win because we can give expanded mystery shopper offerings and deeper insights using AI”, Mills states, highlighting the potential for differentiation in a highly competitive space.

Conclusion: When considering LLMs, think beyond RAG – think workflows

Enterprises can embed LLMs in business workflows to improve efficiency, enable higher quality work, and boost customer satisfaction. AI assistants can leverage frameworks other than RAG, and it’s important to consider the right approach to building an LLM solution. In this use case, by automating preliminary review and implementing consistency checks by the AI, report reviewers are enabled to spend more time focusing on quality and educating shoppers on better reporting. These advantages become differentiators in the market and improve customer satisfaction, proving the value of AI solutions.

For organizations looking to leverage AI in their customer experience programs, the time to act is now. The competitive advantage gained through early adoption of these technologies can create significant market differentiation and deliver lasting value to both the organization and its clients.

Ready to transform your customer experience measurement program with AI? Contact Ironside to learn how we can help you achieve operational excellence and deliver enhanced value to your client: GetInsights@IronsideGroup.com

Rapid Insights with Amazon Q in QuickSight

/by

Now more than ever, having the capacity to quickly access and evaluate data is essential for maintaining a competitive advantage. With reliable data gathering and robust infrastructure being the standard across industries, business leaders must take the next step: maximizing the valuable insights gleaned from this abundance of data. Business intelligence (BI) tools are typically the access point for stakeholders to get their reports and dashboards; however building out and maintaining a robust reporting ecosystem can require significant time and effort from analysts, engineers and BI teams.

Decision-makers have constantly evolving needs but the data required to make informed choices isn’t always readily accessible. Compounding the issue is that building ad-hoc reports demands considerable time and involves numerous back-and-forth with the BI team. This presents significant bottlenecks in time-to-insight. But what if business users could directly access and analyze their data without relying on technical expertise?

What Amazon Q in QuickSight Is

Amazon Q in QuickSight is a generative AI assistant embedded within QuickSight, the Amazon Web Services browser-based cloud Business Intelligence tool. The tool allows users to ask natural language inquiries about data and receive prompt, accurate answers accompanied by instant visualizations. Developers can also leverage natural language questions to create visuals, executive summaries, forecasts and anomaly reports. 

Amazon Q in QuickSight has the potential to significantly reduce time to insight and lighten the load on dashboarding teams. While BI analysts focus on building data sources and critical reports, business users can generate ad-hoc reports in seconds rather than hours. QuickSight operates securely within your Amazon QuickSight account, requiring only clean data sources, well-structured metadata, and well-formulated questions.

What it Is Not

Amazon Q in QuickSight is not a replacement for QuickSight or any other BI tool, nor is it a general-purpose AI virtual assistant. Unlike general-purpose AI assistants, such as Amazon Q Business, which use data from across the organization to answer broad business questions, Amazon Q in QuickSight is designed for narrow, data-specific inquiries. It works best with business-specific questions related to datasets that have been prepared for natural language queries, providing focused, data-driven answers tailored to the user’s needs

Leveraging Amazon Q in QuickSight

For Chief Data Officers

Empower your teams to explore data independently with Amazon Q in QuickSight. This tool allows everyone in the organization to ask questions and gain insights without relying on data analysts for every query. By democratizing data access, you can foster a culture where business decisions are quicker and more informed. This increased autonomy also frees up your data teams to focus on complex analyses and strategic initiatives, driving greater value across the organization.

For Business Intelligence Managers

Streamline your team’s workload by routing frequently asked data questions through Amazon Q in QuickSight. This approach significantly reduces the time spent on repetitive data requests and ad-hoc reports, allowing your team to focus on higher-value tasks. For example, you could create a Q Topic dedicated to quality assurance on dimensional data or for validating database metrics without writing SQL. By implementing Amazon Q in QuickSight, you not only boost your team’s productivity but enhance the efficiency of teams you work with as well.

For Analysts

Use Amazon Q in QuickSight to validate hypotheses and gather quick insights during exploratory analysis. Save time on running queries by asking data questions right in the Amazon Q in QuickSight interface. The tool also facilitates quickly switching between datasets, as all topics are available and can be accessed in any part of QuickSight. Amazon Q in QuickSight also supports building dashboard visuals and writing calculations directly in the Q interface, further accelerating report development.

Creating a Topic

Using Amazon Q in QuickSight is simple by design. Once enabled, the tool appears as an icon within QuickSight, which the user can click, ask a question about data and receive an answer.

To successfully implement Amazon Q in QuickSight, data topics must first be set up, which requires thoughtful planning.

Start Asking the Right Questions

Before even choosing the right dataset to answer questions, it’s important to establish a clear business case and determine what questions stakeholders need answered. This process is iterative and adaptable.

Start by engaging with your key stakeholders to understand their pain points and goals. What insights are driving business decisions? Which reports are they using frequently? Which are they rarely using? How much time is being spent on ad-hoc reporting? The insights gained from this process is invaluable, and ensures the stage is set for Amazon Q in QuickSight implementation.

Building Topics

Topics for Amazon Q in QuickSight are organized sets of data designed to give context for the AI, allowing users to ask questions and receive answers in seconds. When an end user interacts with Q, they will do so by selecting a topic within the QuickSight interface and asking questions related to that topic.

To create a topic, you must first select the datasets the topic will use, and manipulate the metadata to fit the chosen business case to make it user-friendly.  

Identify Jargon

During this process, note down any business-specific language used by stakeholders. When creating a topic, you can give data fields synonyms. This helps Amazon Q in QuickSight understand questions and correctly identify the relevant field to use in its analysis. If a stakeholder asks a question using business jargon known to them, they should still get an answer they expect. Organizations that maintain a glossary/dictionary of terms for their business can use this document to further speed up adoption of Amazon Q in QuickSight by their stakeholders.

Topics: Broad or Narrow?

Should your topic be about sales across the organization, or only one product? Will it contain sales data, inventory data, or both? These are critical decisions to make when setting up your topics in Amazon Q in QuickSight.

Topics encompassing a broad range of target data can answer diverse questions and provide a comprehensive view of the business. They are useful for high-level reporting and for users who require a general overview.

Topics that are narrower in focus might include sales data, customer accounts, key financials or other limited scope data. These focused topics offer more targeted insights, making it easier for end users to get relevant answers from Amazon Q in QuickSight quickly.

Your exact topic setup can vary widely pending the business need defined at the outset, data available and audience for the topic. One helpful feature offered by QuickSight that helps with this is Row Level Security available natively within topics. When there is Row Level Security enforced on a QuickSight dataset, and a topic is built using that dataset, the user will only receive insights about data they have access to, with no additional setup required.

Choosing the Right Data

Once you have a clear understanding of common stakeholder data questions, pain points, and how they are talking about their data, you will then naturally find the data sources containing the information best equipped to answer them. The data you choose will be the bedrock of your Amazon Q in QuickSight solution.

Setting Up Topics

QuickSight will perform an initial ‘cleaning’ of your data on its own, assigning common synonyms to field names (e.g., a field called ‘Cost of Goods Sold’ will be given the synonym ‘COGS’ automatically). You will want to double check this for accuracy, as it can make mistakes.

Once data is selected, QuickSight will prompt you to ‘Start Review’ of the topic data. This is almost always your immediate first step. Proper data cleaning and preparation are crucial steps to ensure that Q in QuickSight provides accurate, actionable insights that align with your business goals.

Field Selection

Review each field to ensure that only relevant and high-quality data is included. Consider which fields are adding value to the analysis, and which are adding noise or aren’t useful for the analysis. Include and exclude fields as needed.

Synonyms

Add synonyms to fields to accommodate different terminologies used by your business users. This will make the topic (and results) more user-friendly and intuitive.

Data Roles and Formatting

Define roles (such as dates, measures, dimensions, etc.) and apply consistent formatting. You can manually set fields to be formatted as Currency, which will maintain the clarity and overall quality of the insights.

Data Aggregations

Set up the data aggregations of measures to set up meaningful summarizations. You can set default aggregations (e.g., a ‘Sales’ field can be set to SUM or AVERAGE automatically) and disallow other aggregations that either don’t make sense or are potentially sensitive.

Testing, Deployment and Beyond

Amazon Q in QuickSight encourages continued evaluation and performance monitoring after a topic is deployed to end users. For each topic, you can see how many questions have been asked and what they are, and view user feedback of their responses. Each time a user asks a question, they have the option to give the response a thumbs up or thumbs down, and give long text feedback.

End users should be highly encouraged to interact with the results they get from Amazon Q. Their input is extremely valuable and can be used to determine what is working well with the topic and what could use improvement. Additionally, you can see how many questions have been asked in total, what portion of those questions were answerable, and what portion weren’t. Use these metrics to assess engagement with the solution and calibrate or update the topics accordingly.

Listening to Feedback

From the outset, deploying Amazon Q in QuickSight works best as a collaborative process. Integrating it into your BI ecosystem is best done by working closely with the target end users at all stages of development, from ideation to deployment. Their expectations and usage of the tool will be the biggest variable impacting the success of the project, and their continued engagement after deployment will ensure issues are identified, fixed and maintained.

Negative feedback from users should always be addressed and corrected quickly. If end users find many of their questions unanswerable or are consistently getting unexpected results from the topic, it can lead to frustration and abandonment of the tool. Remember, the value of Q in QuickSight is being able to get the information you need quickly and easily. If users are using synonyms that the development team had not considered, if their queries are too complex, or irrelevant to the topic, they will have trouble getting the results they are looking for.

A great way to start users out using Amazon Q in QuickSight is with Suggested Questions. These are questions shown to users by default when they open a topic in the Q in QuickSight interface within the QuickSight console. These can be validated by the developers of the solution. Questions can be manually validated by the developer of the topic and shown to end users by default. These can also be re-validated periodically as the data refreshes. End users are shown the last validation time.

Setting Expectations

Given the rise in popularity of other AI-powered tools, it’s important to set user expectations about what Amazon Q in QuickSight is and isn’t useful for. It is designed for quick, high-level insights while also giving a user the ability to dig deep to answer specific questions about their data. Questions posed to a Topic should always be pertinent to the data in the topic.

Amazon Q in QuickSight is not a replacement for mission-critical dashboards, nor is it a general purpose, world-wise AI virtual assistant. Situations involving intricate calculations or carefully formatted reports are better carried out in a QuickSight Analysis. Additionally, while Amazon Q in QuickSight streamlines the querying process, it still requires well-prepared and clean data to function effectively.

Final Thoughts

Amazon Q in QuickSight is an AI solution completely integrated into a full-fledged BI tool. It is secure, and enables end users to perform their own ad-hoc reporting done at their convenience. By empowering users to query data directly and intuitively, Amazon Q in QuickSight reduces the load placed on technical teams, accelerates decision-making processes, and fosters a more data-driven culture within the organization. These make Amazon Q in QuickSight a valuable asset for any organization looking to enhance its reporting capabilities and drive positive business outcomes.. 

AWS Summit New York 2024: Transforming Cloud Innovation and AI Solutions

/by

The AWS Summit New York 2024 was an exhilarating event to showcase cloud innovation, AI advancements, and industry best practices. At this action-packed day hosted at the Jacob K. Javits Convention Center, this year’s Summit brought together thousands of professionals, technology enthusiasts, and AWS experts to explore how cutting-edge AWS technologies can be used to revolutionize industries and empower businesses.

At this year’s Summit, over 170 sessions were offered covering a wide range of topics and technical depth, ranging from level 100 (foundational), level 200 (intermediate), level 300 (advanced), and level 400 (expert). Within these sessions, many AWS experts, builders, customers, and partners shared their insights on numerous topics such as generative AI, analytics, machine learning, industry specific solutions, and many more. Individuals were able to customize their own agenda ahead of time and choose from lecture-style presentations, peer-led discussions, and explore the Expo to learn about the numerous advancements of AWS technologies and deepen understanding of best practices. Dr. Matt Wood, VP for AI Products, AWS, hosted the keynote session to unveil the latest launches and technical innovations from AWS and demonstrate products and real-world success stories from AWS customers.

Below is a detailed look at some of my key takeaways and trends that summarizes this year’s Summit:

1. Amazon Bedrock

Stemming from the heavy emphasis on generative AI and its capabilities, one of the most exciting announcements from the Summit was the introduction of new capabilities in Amazon Bedrock. Amazon Bedrock is AWS’s relatively new service designed to simplify the creation of AI applications. The service provides access to pre-trained foundation models from leading AI providers, and enables businesses to build, deploy, and scale AI-driven solutions without deep expertise and extensive effort. In addition, the many key features of Amazon Bedrock allow users and businesses to build innovative AI solutions effectively and efficiently while ensuring scalability and compliance. The fundamental idea of this service is to revolutionize how companies develop and deploy generative AI applications, making it easier to integrate cutting-edge technology into existing workflow while significantly reducing computational costs. 

At this year’s Summit, additional features of Amazon Bedrock were introduced to enhance company knowledge bases with new Amazon Bedrock connectors for Confluence, Salesforce, SharePoint, and web domains. In doing so, companies can empower RAG models with contextual data for more accurate and relevant responses. 

Lastly, Guard Rails and Guard Rails API were introduced for Amazon Bedrock to contribute to the following:

  • Bring a consistent level of AI safety across all applications
  • Block undesirable topics in generative AI applications
  • Filter harmful content based on responsible AI policies
  • Redact sensitive information (PII) to protect privacy
  • Block inappropriate content with a custom word filter
  • Detect hallucinations in model responses using contextual grounding checks

Businesses and customers can apply safeguards to generative AI applications even if those models are hosted outside of AWS infrastructure. It is estimated that up to 85% of harmful content can be reduced with custom Guardrails.

2. Fannie Mae’s Data Science Platform

One of the first sessions that I attended was Fannie Mae’s presentation on their data science platform. The focus was on how Fannie Mae overcame traditional data management challenges through innovative solutions. Data scientists at Fannie Mae were responsible for exploring internal and external datasets, including sensitive data to develop and train models, create reports and new datasets, deploy models, and share insights. Before the utilization of AI, Fannie Mae’s data scientists struggled with data access (mostly personally identifiable information), governance, and operationalization. In addition, underwriting analysts spent significant time extracting structured data from unstructured documents. On average, each analyst spent 5 hours on every document, with over 8,000 underwriting documents per year. The challenge of inefficient manual document analysis was also resolved by the utilization of AI.

By leveraging Large Language Models (LLMs) and ontologies, Fannie Mae developed a knowledge extraction system that significantly reduced manual effort. Tools like Amazon Bedrock, Claude 3 Sonnet, Amazon Neptune, LangChain, and Amazon OpenSearch Service played a crucial role in this transformation. The use of AI has generated a potential savings of over 32,000 hours annually and improvements in accuracy, compliance, and scalability of underwriting analysis for Fannie Mae.

Such efficiency and savings generated by the use of LLMs and ontologies is simply fascinating. This is a great reflection on how companies of all sectors can utilize the diverse capabilities of AI and customizable machine learning models to generate value.

3. IBM WatsonX & AWS: Scale Gen AI Impact with Trusted Data

Generative AI was a major theme at the Summit, and IBM WatsonX and AWS highlighted their collaborative efforts to expand the impact of this technology. The WatsonX suite offers tools like Watsonx.ai for model development, Watsonx.data for scaling AI workloads, and Watsonx.governance for ensuring responsible AI practices. This partnership brings a shift towards more open, targeted, and cost-effective generative AI solutions, while offering superior price-performance at less than 60% of the traditional costs.

4. Advancing AI and Cloud Solutions

Another key topic of the Summit was Innovating with Generative AI on AWS. This topic highlights how businesses can focus on performance, cost-efficiency, and ethical responsibilities in AI development. Many strategies were discussed for creating new customer experiences, boosting productivity, and optimizing business processes through generative AI.

Some of the key techniques included Retrieval Augmented Generation (RAG) for combining new and existing information, fine-tuning of AI models, and pre-training to enhance AI capabilities. The session emphasized the importance of accessible and high-quality data as the foundation for AI success, so that businesses can utilize generative AI to its maximum potential to drive innovation and create value. By using services designed to enable innovation and scale, businesses are able to measure and track value and ROI while optimizing for cost, latency, and accuracy needs. In addition, businesses can manage risk, maintain trust, and build with compliance and governance.

5. Boosting Employee Productivity with AI Agents

Another highlight was the exploration of AI agents powered by Amazon Q. With Amazon Q, businesses can design these AI agents to integrate seamlessly with tools like Slack, Microsoft Teams,  and other AWS-supported data sources to enhance employee productivity. These AI agents can improve efficiency across teams and organizations by streamlining data interactions and automating repetitive tasks. A demo of how to connect the Slack instance to Amazon Q and deploy it into the Slack workspace showed the simplicity of the whole process and how quick Amazon Q can generate value for an organization.

6. Building a Strong Data Foundation for Generative AI

A central theme at the Summit was the importance of a solid data foundation for successful generative AI initiatives. AWS demonstrated how businesses can harness structured and unstructured data through various tools and services. Key components of this foundation include:

  • Data Storage: Managing structured and unstructured data using SQL, NoSQL, and graph databases
  • Data Analytics: Utilizing data lakes for search, streaming, and interactive analytics
  • Vector Embeddings: Tokenizing and storing data for semantic similarity searches
  • Data Integration: Combining data from different sources using tools like AWS Glue and Amazon DataZone.

7. Governance and Compliance in the Cloud

Governance and compliance were also significant topics, with AWS highlighting how organizations can manage data securely and efficiently. Enterprise customers look for democratized data tools with built-in governance to discover, understand, and access data across organizations, with the ability for multiple personas to collaborate on the same data problems. In addition, easy-to-use and easy-to-access analytics and BI tools are crucial for value creation. The Summit showcased services like AWS IAM, Amazon Cognito, AWS Lake Formation, and Amazon S3 for data management, access control, and auditing. These tools help ensure that cloud operations are compliant with regulations and best practices

8. The Future of Generative AI

Lastly, the Summit concluded with a discussion on the future of generative AI. The evolution of AI agents such as Ninjatech.AI, multimodal models, and new regulations were some of the topics that were discussed. The session also explored the balance between value and feasibility in AI projects. It is crucial to identify the value generated from productivity, experience, and revenue, but also focus on the need for innovation that is both effective and sustainable.

The AWS Summit New York 2024 highlighted the latest advancements in cloud technology and AI. One of the major releases, Amazon Bedrock, allows businesses to build, deploy, and scale AI-driven solutions without extensive expertise and effort. This promotes businesses to focus more on performance, cost, and ethical responsibilities with gen AI.

The Summit offered valuable insights and tools for businesses looking to leverage cloud computing for innovation and efficiency. Many case studies were showcased to further support the adoption of generative AI in businesses of all sectors and instances where generative AI can create value for all aspects of the business. The sense of urgency to adopt gen AI has doubled since last year, and the emphasis to build a solid data foundation for successful generative AI initiatives has never been greater. The many new innovations simplifies the process for businesses to leverage data to create and differentiate generative AI applications, and create new value for customers and the business. The phrase “Your data is the differentiator” should be remembered as businesses navigate through the AI journey. 

Overall, the AWS Summit provided a comprehensive look at how AWS is shaping the future of technology. With a strong emphasis on AI and machine learning advancements, security enhancements, and sustainability efforts, the future has never looked so bright for businesses, developers, and consumers.