Proper conflict-free coloring is an intermediate notion between proper coloring of a graph and proper coloring of its square. It is a proper coloring such that for every non-isolated vertex, there exists a color appearing exactly once in its (open) neighborhood. Typical examples of graphs with large proper conflict-free chromatic number include graphs with large chromatic number and bipartite graphs isomorphic to the $1$-subdivision of graphs with large chromatic number. In this paper, we prove two rough converse statements that hold even in the list-coloring setting. The first is for sparse graphs: for every graph $H$, there exists an integer $c_H$ such that every graph with no subdivision of $H$ is (properly) conflict-free $c_H$-choosable. The second applies to dense graphs: every graph with large conflict-free choice number either contains a large complete graph as an odd minor or contains a bipartite induced subgraph that has large conflict-free choice number. These give two incomparable (partial) answers of a question of Caro, Petruševski and Škrekovski. We also prove quantitatively better bounds for minor-closed families, implying some known results about proper conflict-free coloring and odd coloring in the literature. Moreover, we prove that every graph with layered treewidth at most $w$ is (properly) conflict-free $(8w-1)$-choosable. This result applies to $(g,k)$-planar graphs, which are graphs whose coloring problems have attracted attention recently.