In this post, I mainly focus on introducing Gelfand pairs of finite groups.

### Representations of finite groups

Let’s recall some concepts. Let $G$ be a finite group. A (complex) representation of $G$ is a pair $(\rho, V)$ where $V$ is a vector space over $\C$ and $\rho: G \to GL(V)$ is a group homomorphism. Let $(\rho_1, V_1)$ and $(\rho_2, V_2)$ be representations of $G$, a $G$-morphism, or simply a morphism, between these two representations is a linear map $\varphi: V_1 \to V_2$ such that for any $g \in G$ and $v \in V_1$, $\varphi(\rho_1(g)v) = \rho_2(g)\varphi(v).$ Then $\mathrm{Hom}_G(\rho_1, \rho_2)$, the set of morphisms between $\rho_1$ and $\rho_2$, is a vector space over $\C$. A representation $(\rho, V)$ is irreducible, if there is no proper subspace $W \subset V$, such that $\rho(g)w \in W$ for any $g \in G$ and $w \in W$. Between irreducible representations, we have Schur’s lemma.

Theorem 1 (Schur’s lemma). Let $\rho_1$ and $\rho_2$ be irreducible representations of $G$. Then $\begin{equation*} \mathrm{Hom}_G(\rho_1, \rho_2) \cong \left\{ \begin{array}{ll} \C, & \text{ if } \rho_1 \cong \rho_2; \\ 0, & \text{ otherwise.} \end{array} \right. \end{equation*}$

It’s well known that $G$ is semisimple, i.e., any representation of $G$ decomposes into a direct sum of irreducible representations. A representation $(\rho, V)$ of $G$ is called multiplicity free, if any irreducible representation appearing in the decomposition of $(\rho, V)$ appears exactly once, up to isomorphism.

Let $H$ be a subgroup of $G$. If $(\rho, V)$ is a representation of $G$, we can certainly restrict the group homomorphism $\rho: G \to GL(V)$ to $H$ so that we can regard $(\rho, V)$ as a representation of $H$. This is called the restriction of representations from $G$ to $H$, and the restricted representation of $\rho$ will be denoted as $r^G_H(\rho)$. Conversely, if $(\pi, W)$ is a representation of $H$, we can build a representation $(\rho, V)$ of $G$ out of $\pi$, which is the induced representation, as follow.

$\newcommand{\slash}{{\bf /}}$

Let $V$ be a space of functions defined by $V = \{f: G \to W: f(hg) = \pi(h)f(g)\}.$ The group homomorphism $\rho: G \to GL(V)$ is given by $(\rho(g)f)(g^\prime) = f(g^\prime g).$ We use the notations $\rho = i_H^G(\pi)$ and $V = i_H^G(W)$ for induced representations. Restrictions and inductions of representations are related by Frobenius reciprocity theorem.

Theorem 2 (Frobenius reciprocity). Let $(\rho, V)$ be representation of $G$ and let $(\pi, W)$ be representation of $H$. Then there is a canonical linear isomorphism $L: \mathrm{Hom}_G(\rho, i_H^G \pi) \to \mathrm{Hom}_H(r^G_H \rho, \pi)$ given by $L(\varphi)(v) = \varphi(v)(e),$ where $\varphi \in \mathrm{Hom}_G(\rho, i_H^G \pi), v \in V$ and $e$ is the identity of $G$. Moreover, the inverse $L^{-1}: \mathrm{Hom}_H(r^G_H \rho, \pi) \to \mathrm{Hom}_G(\rho, i_H^G \pi)$ is given by $L^{-1}(\psi)(v)(g) = \psi(\rho(g)v),$ where $\psi \in \mathrm{Hom}_H(r^G_H \rho, \pi), v \in V$ and $g \in G$.

Also, we have Mackey’s theorem relating two representations inducing from different representations. Let $H_1$ and $H_2$ be subgroups of $G$. Let $(\pi_1, W_1)$ and $(\pi_2, W_2)$ be representations of $H_1$ and $H_2$, respectively. Consider the space $\mathcal{D}$ of functions $\Delta: G \to \mathrm{Hom}_\C(W_1, W_2)$ such that $\Delta(h_2 g h_1) = \pi_2(h_2) \Delta(g) \pi_1(h_1),$ for any $h_1 \in H_1, h_2 \in H_2$ and $g \in G$.

Theorem 3 (Mackey’s theorem). As vector spaces, $\mathcal{D} \cong \mathrm{Hom}_G(i_{H_1}^G \pi_1, i_{H_2}^G \pi_2).$

**Remark. **If $(\pi_1, W_1) = (\pi_2, W_2)$, implicitly, this implies $H_1 = H_2$. Let $H=H_1=H_2$ and let $\pi=\pi_1=\pi_2$. In this case, for any $\Delta_1, \Delta_2 \in \mathcal{D}$, we can define convolution by $\begin{equation*} \Delta_1 \ast \Delta_2(g) = \sum_{g^\prime \in G} \Delta_1({g^\prime}) \circ \Delta_2({g^\prime}^{-1}g). \end{equation*}$ Then $\mathcal{D}$ becomes a ring. On the other hand, $\mathrm{Hom}_G(i_H^G \pi, i_H^G \pi)$ is naturally a ring by compositions. Under this special situation, the isomorphism in Theorem 3 is actually a ring isomorphism.

### Gelfand pairs with respect to trivial character

Let $C(H \backslash G \slash H)$ be a set of functions $\Delta: G \to \C$ such that $\Delta(h_1 g h_2) = \Delta(g).$ Given $\Delta_1, \Delta_2 \in C(H \backslash G \slash H)$, define the convolution $$$\Delta_1 \ast \Delta_2(g) = \sum_{{g^\prime} \in G} \Delta_1({g^\prime})\Delta_2({g^\prime}^{-1}g). \label{20150928-1}$$$

It’s quite straightforward to check that $\Delta_1 \ast \Delta_2 \in C(H \backslash G \slash H)$. Then under point-wise addition and convolution, $C(H \backslash G \slash H)$ becomes a ring.

Theorem 4.Let $G$ be a finite group and let $H$ be a subgroup. Let $C(H \backslash G) = \{f:G \to \C: f(hg)=f(g) \, \forall h \in H, g \in G\}.$ The right translation $\rho$ of $G$ on $C(H \backslash G)$ is a representation. The following are equivalent.

(1) Every irreducible representation $V$ of $G$ has at most one $H$-fixed vector, up to scalar multiplication.

(2) $(\rho, C(H \backslash G))$ is multiplicity free.

(3) $C(H \backslash G \slash H)$ is commutative under convolution.

Proof. Suppose statement $1$ and statement $2$ are equivalent. Let $(\chi_0, \C)$ be the trivial representation of $H$, i.e., $\chi_0(h) = id_\C$ for any $h \in H$. Then by the definition of induction, $\rho = i_H^G \chi_0$. $\rho$ is multiplicity free, if and only if for any irreducible representation $\pi$ of $G$, $\pi$ appears in $\rho$ at most one. By Theorem 1, the number of times of $\pi$ in $\rho$ can be calculated by $\dim \mathrm{Hom}_G(\pi, \rho)$. Using Theorem 2, this is the same as $\dim \mathrm{Hom}_H(i_H^G \pi, \chi_0)$, i.e., the dimension of $H$-fixed vectors in $\pi$.

Suppose statement $2$ is equivalent to statement $3$. By Theorem 3 (See also Remark after the theorem), $C(H \backslash G \slash H) \cong \mathrm{Hom}_G(\rho, \rho) \text{ as rings}.$ Using Theorem 1, we can see that $\rho$ is multiplicity free if and only if $\mathrm{Hom}_G(\rho, \rho)$ is commutative, therefore if and only if $C(H \backslash G \slash H)$ is commutative. $\square$

Definition 5.Let $G$ be a finite groups and let $H$ be a subgroup. Then $(G, H)$ is called a Gelfand pair (with respect to $\chi_0$), if $G$ and $H$ satisfy the equivalent statements in Theorem 4.

### Gelfand pairs with respect to general characters

Let’s look back to Theorem 4 and see how it relates to the trivial character $\chi_0$ of $H$. As seen in the proof of Theorem 4, the first statement is equivalent to $\dim \mathrm{Hom}_H(r^G_H \pi, \chi_0) \le 1$ for any irreducible representation $\pi$ of $G$. In the second statement, $(\rho, C(H \backslash G)) = (i_H^G \chi_0, i_H^G \C).$ And in the third statement, we know by Theorem 3 that $C(H \backslash G \slash H) \cong \mathrm{Hom}_G(\rho, \rho) = \mathrm{Hom}_G(i_H^G \chi_0, i_H^G \chi_0).$ Therefore, all statements are actually coming from $\chi_0$ and it’s not surprising that Theorem 4 holds with $\chi_0$ replaced by any character $\chi$ of $H$.

Let $\chi$ be a character of $H$. Define $C(H \backslash G \slash H; \chi)$ to be a space of functions $\Delta: G \to C$ such that $\Delta(h_1 g h_2) = \chi(h_1)\Delta(g)\chi(h_2),$ for any $h_1, h_2 \in H$ and $g \in G$. Then $C(H \backslash G \slash H; \chi)$ becomes a ring under the convolution defined in $\eqref{20150928-1}$. Using Theorem 3, $C(H \backslash G \slash H; \chi) \cong \mathrm{Hom}_G(i_H^G \chi, i_H^G \chi).$

Now following the same proof in Theorem 4, we can prove a similar theorem. Henceforth, we get a more general definition for Gelfand pairs.

Theorem 6.The following are equivalent.

(1) $\dim \mathrm{Hom}_H(r^G_H \pi, \chi) \le 1$, for any irreducible representation $\pi$ of $G$.

(2) $i_H^G \chi$ is multiplicity free.

(3) $C(H \backslash G \slash H; \chi)$ is commutative under the convolution.

Definition 7.Let $G$ be a finite groups and let $H$ be a subgroup. Then $(G, H)$ is called a Gelfand pair with respect to $\chi$, if $G$ and $H$ satisfy the equivalent statements in Theorem 6.

### Gelfand’s Lemma

We use notations as in previous sections. A set bijection $\iota: G \to G$ is called an anti-involution if $\iota(gh) = \iota(h)\iota(g)$, for any $h, g \in G$. Then $\iota$ induces an action on the double coset $H \backslash G \slash H$, which sends $HgH$ to $H\iota(g)H$.

Theorem 8 (Gelfand’s lemma). Let $\iota: G \to G$ be an anti-involution. The ring $C(H \backslash G \slash H; \chi)$ is commutative under convolution if $\iota$ acts trivially on $H \backslash G \slash H$ and $\chi$, i.e., $\iota(r_i)=r_i$ for a set of representatives $\{r_i\}$ of $H \backslash G \slash H$ and $\iota(h) \in H$ with $\chi(h) = \chi(\iota(h))$ for $h \in H$.

Proof. Define an action of $\iota$ on $C(H \backslash G \slash H; \chi)$ by $\iota(\Delta)(g) = \Delta(\iota(g)).$ The action is well defined, because for $h_1, h_2 \in H$, $$$\begin{split} \iota(\Delta)(h_1 g h_2) &= \Delta(\iota(h_2) \iota(g) \iota(h_1)) \\ &= \chi(\iota(h_2)) \Delta(\iota(g)) \chi(\iota(h_1)) \\ &= \chi(h_2) \iota(\Delta)(g) \chi(h_1), \end{split} \label{20150928-2}$$$

so $\iota(\Delta) \in C(H \backslash G \slash H; \chi)$. By an easy calculation, we can see that $\iota$ is an anti-involution on $C(H \backslash G \slash H; \chi)$, $\iota(\Delta_1 \ast \Delta_2) = \iota(\Delta_2) \ast \iota(\Delta_1).$ Moreover, since $\iota$ acts trivial on $H \backslash G \slash H$, a calculation similar to $\eqref{20150928-2}$ shows that $\iota$ is actually the identity map on $C(H \backslash G \slash H; \chi)$. If the identity map is anti-involution, then the ring $C(H \backslash G \slash H; \chi)$ is commutative. $\square$

Remark.Actually, we don’t need $\iota$ to act trivially on all $r_i$. We just need $\iota$ act trivially on those $r_i$ having some $\Delta$ such that $\Delta(r_i) \ne 0$.

The condition that $\iota$ acts trivially on $\chi$ is obtained under either of the following situations.

(1) $\iota$ acts trivially on $H$, i.e., $\iota(h)=h$ for $h \in H$.

(2) $\chi=\chi_0$ and $\iota$ stabilizes $H$, i.e., $\iota(h) \in H$ for $h \in H$.

### An application: Uniqueness of Whittaker models

Let $F$ be a finite field. Let $G=GL(2, F)$ be the invertible $2 \times 2$ matrices, and let $N$ be a subgroup of $G$ defined by $\begin{equation*} N = \left\{ \left(\begin{array}{cc} 1 & x \\ 0 & 1 \end{array}\right) : x \in F \right\}. \end{equation*}$ $N$ is so called the standard nilpotent subgroup of $G$. Let $\psi$ be a nontrivial additive character of $F$, then it can be extended to a character of $N$ given by $\begin{equation*} \psi_N\left(\begin{array}{cc} 1 & x \\ 0 & 1 \end{array}\right) =\psi(x). \end{equation*}$ Then we can induce $\psi_N$ to form a representation $(\rho, \mathcal{W}_\psi) = (i_N^G \psi_N, i_N^G \C)$ of $G$. Given an irreducible representation $(\pi, V)$ of $G$, we say $\pi$ has a Whittaker model with respect to $\psi_N$ if $\pi$ can be embedded into $\mathcal{W}_\psi$. We call the image of this embedding the Whittaker model of $\pi$.

The motivation of Whittaker models is clear: We try to realize an abstract irreducible representation as a concrete space of functions $f: G \to \C$ such that $\begin{equation*} f\left(\left(\begin{array}{cc} 1 & x \\ 0 & 1 \end{array}\right)g\right) =\psi(x)f(g). \end{equation*}$

Given an abstract irreducible representation $(\pi, V)$, there are two immediate questions should be ask:

(1) Does $\pi$ have a Whittaker model with respect to some character $\psi$?

(2) If $\pi$ has a Whittaker model, is it unique?

As for the first question, we need to understand what kinds of irreducible representation of $G$ have. This would be another long story and not related to this post, so I would rather ignore it. But the upshot is that not every irreducible representation has a Whittaker model.

For the second question, it’s the same to ask whether or not $\mathcal{W}_\psi$ is multiplicity free. Therefore, based on Theorem 6, it suffices to answer whether or not $(G, N)$ is a Gelfand pair with respect to $\psi$.

Theorem 9 (Uniqueness of Whittaker models). $(G, N)$ is a Gelfand pairs with respect to any non-trivial character $\psi$. Equivalently, $\mathcal{W}_\psi$ is multiplicity free.

Proof. Under the light of Theorem 8, we just need to find an anti-involution $\iota: G \to G$ such that $\iota$ acts trivially on $N \backslash G \slash N$ and $N$ (see Remark after Theorem 8).

It’s from Bruhat decomposition that the representatives of $N \backslash G \slash N$ are of forms $\begin{equation*} \left(\begin{array}{cc} a & \\ & d \end{array}\right), \left(\begin{array}{cc} & b\\ c & \end{array}\right). \end{equation*}$ Recall that we want to prove $C(N \backslash G \slash N; \psi)$ is commutative as in Theorem 8. Let $\Delta \in C(N \backslash G \slash N; \psi)$, then $\begin{equation*} \begin{split} &\Delta \left(\begin{array}{cc} a & \\ & d \end{array}\right) \\ &= \Delta \left( \left(\begin{array}{cc} 1 & -a\\ & 1 \end{array}\right) \left(\begin{array}{cc} a & \\ & d \end{array}\right) \left(\begin{array}{cc} 1 & d\\ & 1 \end{array}\right) \right) \\ &= \psi(d-a) \Delta \left(\begin{array}{cc} a & \\ & d \end{array}\right) \end{split} \end{equation*}$ Since $\psi$ is non-trivial, $\Delta$ is support on $\left(\begin{array}{cc}a &\\& d \end{array}\right)$ if and only if $a=d$. Therefore, we just need to find an anti-involution $\iota: G \to G$ such that $\iota$ acts trivially on $N$ and on matrices of forms $\begin{equation*} \left(\begin{array}{cc} a & \\ & a \end{array}\right), \left(\begin{array}{cc} & b\\ c & \end{array}\right). \end{equation*}$ Define $\iota: G \to G$ by $\begin{equation*} \iota(g) = \left(\begin{array}{cc} & 1\\ 1 & \end{array}\right) g^T \left(\begin{array}{cc} & 1\\ 1 & \end{array}\right), \end{equation*}$ where $g^T$ is the transpose of $g$. It’s easy to verify that $\iota$ is the desired anti-involution. $\square$