Get text and caret from a textbox in another application

问题

Background information

I am working on a Windows Speech-To-Text application.

When speech has been recognized, the text should be inserted into the textbox that currently has keyboard focus (Think Word/Firefox/other applications). To insert the text I am currently using InputSimulatorPlus.

When inserting the text, it is important the recognized text is formatted to fit the surrounding text, like:

uppercase letter after punctuation
lowercase first letter when it is NOT after punctuation
uppercase first letter on a new line
space after punctuation

And so on

Problem

To be able to format the text I need the text, and the caret position.

Currently I have been using UI Automation NuGet Package with Text Pattern. This works well for all textboxes that support Text Pattern, but alot of programs do not support Text Pattern.

Strategy question: Should I use another approach than UI Automation?

I have noticed that a lot of the applications I want to support do not support Text Pattern, but do support Value Pattern or Legacy IAccessible Pattern(Microsoft Active Accessibility).

I have been looking into using the Value Pattern and can get the text of the textbox, but not the caret position.

using System.Windows.Automation;
using System.Windows.Automation.Text;
...

AutomationElement automationElement = AutomationElement.FocusedElement;
var elements = automationElement.FindAll(TreeScope.Element,
    new AndCondition(
        new PropertyCondition(AutomationElement.HasKeyboardFocusProperty, true),
        new PropertyCondition(AutomationElement.IsValuePatternAvailableProperty, true)));
foreach (AutomationElement element in elements)
{
    if (element.GetCurrentPattern(ValuePattern.Pattern) is ValuePattern valuePattern)
    {
        var text = valuePattern.Current.Value;
        var caret = ? //How to get caret position?
        Console.WriteLine($"Caret: {caret}, Text: {text}");
        return (text,caret);
    }
}

I have also been looking into using the Legacy IAccessible Pattern and can get the text of the textbox, but not the caret position.

using System.Windows.Automation;
using System.Windows.Automation.Text;
...

AutomationElement automationElement = AutomationElement.FocusedElement;
var elements = automationElement.FindAll(TreeScope.Element,
    new AndCondition(
        new PropertyCondition(AutomationElement.HasKeyboardFocusProperty, true),
        new PropertyCondition(AutomationElement.IsLegacyIAccessiblePatternAvailableProperty, true)));
foreach (AutomationElement element in elements)
{
    if (element.GetCurrentPattern(LegacyIAccessiblePattern.Pattern) is LegacyIAccessiblePattern legazyAccessiblePattern)
    {
        var text = legazyAccessiblePattern.Current.Value;
        var caret = ? //How to get caret position?
        Console.WriteLine($"Caret: {caret}, Text: {text}");
        return (text,caret);
    }
}

Main question: How do I get the caret position of a given textbox using UI Automation?

P.S. I know this will never work for all applications, but if it can work for most normal applications, that would be awesome.

回答1:

You don't want ValuePattern, you want IUIAutomationTextPattern2, which supports GetCaretRange.

If you need to fall back to MSAA, it's a lot hackier (and you can't get the text around the caret at all). MSAA just doesn't have anything like the text support that UIAutomation has. If you REALLY REALLY can't use UIAutomation, then your only real option is to use Text Services Framework, which is poorly documented and therefore not widely implemented. If you need Text Services Framework, you may want to look at my blog, which hasn't been updated in quite a while.

For the MSAA caret, start by calling GetGUIThreadInfo. That returns data about a hwndCaret, which is the window which currently contains the caret. (Or you could try using hwndFocus if hwndCaret is NULL.) Using one of those windows, you could do something like:

IAccessible *pAccCaret = NULL;

VARIANT varCaret;
varCaret.vt = VT_I4;
varCaret.lVal = CHILDID_SELF;

if (SUCCEEDED(AccessibleObjectFromWindow(hwnd, OBJID_CARET, IID_IAccessible, (void **)&pAccCaret)))
{
    hr = pAccCaret->accLocation( &rcCaret.left, &rcCaret.top, &rcCaret.right, &rcCaret.bottom, varCaret);

    pAccCaret->Release();
}

来源：https://stackoverflow.com/questions/52592652/get-text-and-caret-from-a-textbox-in-another-application

标签

.net

windows

ui-automation