Edit: In the original question i made some wrong assumptions about how setters work so i modified it to hopefully be more accurate and useful.
So after reading Meleak's answer and finding out that you can have a style within a style via resources what probably comes closest to doing this with a setter is an embedded style to access the icon's opacity. Here i assume the icon to be an image so i use that as the target type, the complete style hence looks like this:
<Style x:Key="MenuItemMouseOverStyle" TargetType="MenuItem">
<Style.Resources>
<Style TargetType="Image">
<Setter Property="Opacity" Value="0.5"/>
</Style>
</Style.Resources>
<Style.Triggers>
<EventTrigger RoutedEvent="MouseEnter">
<BeginStoryboard>
<Storyboard>
<DoubleAnimationUsingKeyFrames Storyboard.TargetProperty="Icon.Opacity">
<EasingDoubleKeyFrame KeyTime="0" Value="0.5"/>
<EasingDoubleKeyFrame KeyTime="0:0:0.3" Value="1"/>
</DoubleAnimationUsingKeyFrames>
</Storyboard>
</BeginStoryboard>
</EventTrigger>
<EventTrigger RoutedEvent="MouseLeave">
<BeginStoryboard>
<Storyboard>
<DoubleAnimationUsingKeyFrames Storyboard.TargetProperty="Icon.Opacity">
<EasingDoubleKeyFrame KeyTime="0" Value="1"/>
<EasingDoubleKeyFrame KeyTime="0:0:0.3" Value="0.5"/>
</DoubleAnimationUsingKeyFrames>
</Storyboard>
</BeginStoryboard>
</EventTrigger>
</Style.Triggers>
</Style>
The only problem with this is that it does not actually set the Icon.Opacity
but the opacity of all images that may occur within the MenuItem.
I don't think you can access a Property of a Property like that so the casting itself isn't the problem. Even if Icon was of Type Image that still wouldn't work. You can try with Backgrounds Opacity for a Grid for example. Background is a Dependency Property for Grid and Opacity is a Dependency Property for Brush but the following line won't work
<Grid Background.Opacity="0.8"/>
You'll get an error saying
The attachable property 'Opacity' was not found in type 'Background'.
You would have to set this in the Background itself like this
<Grid>
<Grid.Background>
<SolidColorBrush Opacity="0.8"/>
</Grid.Background>
</Grid>
So what this means as when you do something like this
<Grid TextBlock.Foreground="Red">
<TextBlock Text="Test"/>
</Grid>
you're actually using the Attached Property Foreground for TextBlock.
Image doesn't have an Attached Property called Opacity so you can't do this either
<MenuItem Image.Opacity="0.8" />
Another workaround besides the one you're already doing is to use something like this (top-most MenuItem or wherever you want to use it).
<MenuItem x:Name="topMenuItem"
...>
<MenuItem.Resources>
<Style TargetType="Image">
<Setter Property="Opacity" Value="0.5"/>
</Style>
</MenuItem.Resources>
<!-- ... -->
</MenuItem>